2026-02-20T16:24:32.123 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-20T16:24:32.138 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T16:24:32.215 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:06:38-rados-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60841 branch: wip-pdonnell-testing-20260219.182737-tentacle description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '60841' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-20_16:06:38-rados-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20260219.182737-tentacle ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f7870454149f22574086dd2ff55d050585136531 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f7870454149f22574086dd2ff55d050585136531 workunit: branch: wip-pdonnell-testing-20260219.182737-tentacle sha1: f7870454149f22574086dd2ff55d050585136531 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8419 sha1: f7870454149f22574086dd2ff55d050585136531 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-pdonnell-testing-20260219.182737-tentacle suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f7870454149f22574086dd2ff55d050585136531 targets: trial013.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNQsHv8lbyGid1/PfNz+pbJ96xxzXNx88iQ02sZDz+rfvmegQZBxSgkzDQ59WvYN1ZC5TLfFoI6pW5isCf0rwmA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-20_16:06:38 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-20T16:24:32.215 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa; will attempt to use it 2026-02-20T16:24:32.215 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa/tasks 2026-02-20T16:24:32.215 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-20T16:24:32.217 INFO:teuthology.task.internal:Checking packages... 2026-02-20T16:24:32.217 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f7870454149f22574086dd2ff55d050585136531' 2026-02-20T16:24:32.217 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-20T16:24:32.217 INFO:teuthology.packaging:ref: None 2026-02-20T16:24:32.217 INFO:teuthology.packaging:tag: None 2026-02-20T16:24:32.217 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20260219.182737-tentacle 2026-02-20T16:24:32.217 INFO:teuthology.packaging:sha1: f7870454149f22574086dd2ff55d050585136531 2026-02-20T16:24:32.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20260219.182737-tentacle 2026-02-20T16:24:32.337 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-677-gf7870454-1jammy 2026-02-20T16:24:32.338 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-20T16:24:32.353 INFO:teuthology.task.internal:no buildpackages task found 2026-02-20T16:24:32.354 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-20T16:24:32.369 INFO:teuthology.task.internal:Saving configuration 2026-02-20T16:24:32.379 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-20T16:24:32.391 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-20T16:24:32.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial013.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:06:38-rados-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60841', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 16:20:53.115126', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKFHekBKERhwZ0UVpVJKFNQjymibEdDxR5dBx3QVKIe1'} 2026-02-20T16:24:32.477 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-20T16:24:32.489 INFO:teuthology.task.internal:roles: ubuntu@trial013.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-20T16:24:32.489 INFO:teuthology.run_tasks:Running task console_log... 2026-02-20T16:24:32.816 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x70795a8eac00>, signals=[15]) 2026-02-20T16:24:32.816 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-20T16:24:32.829 INFO:teuthology.task.internal:Opening connections... 2026-02-20T16:24:32.829 DEBUG:teuthology.task.internal:connecting to ubuntu@trial013.front.sepia.ceph.com 2026-02-20T16:24:32.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T16:24:32.892 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-20T16:24:32.903 DEBUG:teuthology.orchestra.run.trial013:> uname -m 2026-02-20T16:24:32.971 INFO:teuthology.orchestra.run.trial013.stdout:x86_64 2026-02-20T16:24:32.971 DEBUG:teuthology.orchestra.run.trial013:> cat /etc/os-release 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:NAME="Ubuntu" 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:VERSION_ID="22.04" 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:VERSION_CODENAME=jammy 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:ID=ubuntu 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:ID_LIKE=debian 2026-02-20T16:24:33.014 INFO:teuthology.orchestra.run.trial013.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-20T16:24:33.015 INFO:teuthology.orchestra.run.trial013.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-20T16:24:33.015 INFO:teuthology.orchestra.run.trial013.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-20T16:24:33.015 INFO:teuthology.orchestra.run.trial013.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-20T16:24:33.015 INFO:teuthology.orchestra.run.trial013.stdout:UBUNTU_CODENAME=jammy 2026-02-20T16:24:33.015 INFO:teuthology.lock.ops:Updating trial013.front.sepia.ceph.com on lock server 2026-02-20T16:24:33.217 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-20T16:24:33.233 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-20T16:24:33.244 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-20T16:24:33.245 DEBUG:teuthology.orchestra.run.trial013:> test '!' -e /home/ubuntu/cephtest 2026-02-20T16:24:33.249 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-20T16:24:33.261 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-20T16:24:33.262 DEBUG:teuthology.orchestra.run.trial013:> test -z $(ls -A /var/lib/ceph) 2026-02-20T16:24:33.301 INFO:teuthology.orchestra.run.trial013.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-20T16:24:33.302 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-20T16:24:33.412 INFO:teuthology.run_tasks:Running task kernel... 2026-02-20T16:24:33.433 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-20T16:24:33.433 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-20T16:24:33.433 DEBUG:teuthology.orchestra.run.trial013:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T16:24:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T16:24:33.436 DEBUG:teuthology.orchestra.run.trial013:> uname -r 2026-02-20T16:24:33.487 INFO:teuthology.orchestra.run.trial013.stdout:5.15.0-170-generic 2026-02-20T16:24:33.488 INFO:teuthology.task.kernel:Running kernel on trial013: 5.15.0-170-generic 2026-02-20T16:24:33.488 DEBUG:teuthology.orchestra.run.trial013:> sudo apt-get clean 2026-02-20T16:24:33.546 DEBUG:teuthology.orchestra.run.trial013:> sudo apt-get update 2026-02-20T16:24:34.059 INFO:teuthology.orchestra.run.trial013.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T16:24:34.153 INFO:teuthology.orchestra.run.trial013.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-20T16:24:34.506 INFO:teuthology.orchestra.run.trial013.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-20T16:24:34.661 INFO:teuthology.orchestra.run.trial013.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-20T16:24:34.807 INFO:teuthology.orchestra.run.trial013.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-20T16:24:35.223 INFO:teuthology.orchestra.run.trial013.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-20T16:24:35.236 INFO:teuthology.orchestra.run.trial013.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-20T16:24:35.237 INFO:teuthology.orchestra.run.trial013.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-20T16:24:35.273 INFO:teuthology.orchestra.run.trial013.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-20T16:24:35.280 INFO:teuthology.orchestra.run.trial013.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-20T16:24:35.281 INFO:teuthology.orchestra.run.trial013.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-20T16:24:35.442 INFO:teuthology.orchestra.run.trial013.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-20T16:24:35.467 INFO:teuthology.orchestra.run.trial013.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-20T16:24:35.556 INFO:teuthology.orchestra.run.trial013.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-20T16:24:35.568 INFO:teuthology.orchestra.run.trial013.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-20T16:24:35.568 INFO:teuthology.orchestra.run.trial013.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-20T16:24:35.604 INFO:teuthology.orchestra.run.trial013.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-20T16:24:35.604 INFO:teuthology.orchestra.run.trial013.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-20T16:24:35.756 INFO:teuthology.orchestra.run.trial013.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-20T16:24:39.193 INFO:teuthology.orchestra.run.trial013.stdout:Fetched 22.6 MB in 3s (7625 kB/s) 2026-02-20T16:24:39.715 INFO:teuthology.orchestra.run.trial013.stdout:Reading package lists... 2026-02-20T16:24:39.728 DEBUG:teuthology.orchestra.run.trial013:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-20T16:24:39.760 INFO:teuthology.orchestra.run.trial013.stdout:Reading package lists... 2026-02-20T16:24:39.867 INFO:teuthology.orchestra.run.trial013.stdout:Building dependency tree... 2026-02-20T16:24:39.868 INFO:teuthology.orchestra.run.trial013.stdout:Reading state information... 2026-02-20T16:24:39.952 INFO:teuthology.orchestra.run.trial013.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout:linux-image-generic set to manually installed. 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout: sg3-utils-udev 2026-02-20T16:24:39.953 INFO:teuthology.orchestra.run.trial013.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T16:24:39.971 INFO:teuthology.orchestra.run.trial013.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T16:24:39.973 DEBUG:teuthology.orchestra.run.trial013:> dpkg -s linux-image-generic 2026-02-20T16:24:40.024 INFO:teuthology.orchestra.run.trial013.stdout:Package: linux-image-generic 2026-02-20T16:24:40.024 INFO:teuthology.orchestra.run.trial013.stdout:Status: install ok installed 2026-02-20T16:24:40.024 INFO:teuthology.orchestra.run.trial013.stdout:Priority: optional 2026-02-20T16:24:40.024 INFO:teuthology.orchestra.run.trial013.stdout:Section: kernel 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Installed-Size: 21 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Maintainer: Ubuntu Kernel Team 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Architecture: amd64 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Source: linux-meta 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Version: 5.15.0.170.159 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Recommends: thermald 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout:Description: Generic Linux kernel image 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout: This package will always depend on the latest generic kernel image 2026-02-20T16:24:40.025 INFO:teuthology.orchestra.run.trial013.stdout: available. 2026-02-20T16:24:40.025 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T16:24:40.025 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-20T16:24:40.025 DEBUG:teuthology.orchestra.run.trial013:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T16:24:40.075 INFO:teuthology.orchestra.run.trial013.stdout:ttyS1 2026-02-20T16:24:40.094 DEBUG:teuthology.parallel:result is None 2026-02-20T16:24:40.094 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-20T16:24:40.106 INFO:teuthology.task.internal:Creating test directory... 2026-02-20T16:24:40.106 DEBUG:teuthology.orchestra.run.trial013:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T16:24:40.139 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-20T16:24:40.151 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-20T16:24:40.167 INFO:teuthology.task.internal:Creating archive directory... 2026-02-20T16:24:40.167 DEBUG:teuthology.orchestra.run.trial013:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T16:24:40.188 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-20T16:24:40.200 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-20T16:24:40.201 DEBUG:teuthology.orchestra.run.trial013:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T16:24:40.236 INFO:teuthology.orchestra.run.trial013.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:24:40.243 INFO:teuthology.orchestra.run.trial013.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:24:40.244 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-20T16:24:40.256 INFO:teuthology.task.internal:Configuring sudo... 2026-02-20T16:24:40.257 DEBUG:teuthology.orchestra.run.trial013:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T16:24:40.296 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-20T16:24:40.312 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-20T16:24:40.312 DEBUG:teuthology.orchestra.run.trial013:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T16:24:40.343 DEBUG:teuthology.orchestra.run.trial013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T16:24:40.392 DEBUG:teuthology.orchestra.run.trial013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T16:24:40.435 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:24:40.436 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T16:24:40.484 DEBUG:teuthology.orchestra.run.trial013:> sudo service rsyslog restart 2026-02-20T16:24:40.541 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-20T16:24:40.554 INFO:teuthology.task.internal:Starting timer... 2026-02-20T16:24:40.554 INFO:teuthology.run_tasks:Running task pcp... 2026-02-20T16:24:40.575 INFO:teuthology.run_tasks:Running task selinux... 2026-02-20T16:24:40.596 DEBUG:teuthology.task.selinux:Excluding trial013: OS 'ubuntu' does not support SELinux 2026-02-20T16:24:40.596 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-20T16:24:40.596 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-20T16:24:40.596 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-20T16:24:40.596 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-20T16:24:40.611 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-20T16:24:40.614 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-20T16:24:40.727 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-20T16:24:40.738 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-20T16:24:40.739 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial013.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-20T16:27:06.774 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial013.front.sepia.ceph.com')] 2026-02-20T16:27:06.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial013.front.sepia.ceph.com' 2026-02-20T16:27:06.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T16:27:06.865 DEBUG:teuthology.orchestra.run.trial013:> true 2026-02-20T16:27:07.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial013.front.sepia.ceph.com' 2026-02-20T16:27:07.115 INFO:teuthology.run_tasks:Running task clock... 2026-02-20T16:27:07.131 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-20T16:27:07.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T16:27:07.132 DEBUG:teuthology.orchestra.run.trial013:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Command line: ntpd -gq 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: ---------------------------------------------------- 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: ntp-4 is maintained by Network Time Foundation, 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: corporation. Support and training for ntp-4 are 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: available at https://www.nwtime.org/support 2026-02-20T16:27:07.176 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: ---------------------------------------------------- 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: proto: precision = 0.020 usec (-25) 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: basedate set to 2022-02-04 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: gps base set to 2022-02-06 (week 2196) 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: restrict ::: KOD does nothing without LIMITED. 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen and drop on 0 v6wildcard [::]:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen normally on 2 lo 127.0.0.1:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen normally on 3 enp1s0f0 10.20.193.13:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen normally on 4 lo [::1]:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6010%2]:123 2026-02-20T16:27:07.177 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:07 ntpd[9414]: Listening on routing socket on fd #22 for interface updates 2026-02-20T16:27:09.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:09.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:09.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:09.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:09.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:09.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:09 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:10.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:10.177 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:10.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:10.207 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:10.207 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:10.207 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:10 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:11.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:11.177 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:11.250 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:11.250 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:11.250 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:11.250 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:11 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.207 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:12.207 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:12.215 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.215 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.215 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:12.216 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:12.241 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:12.241 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:12.241 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:12.241 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:12 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:13.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:13 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:13.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:13 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:14.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:14.214 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.214 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.214 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:14.214 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:14.242 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:14.242 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:14.242 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:14.242 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:14 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:15.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:15.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:15.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:15.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:15.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:15.211 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:15 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:16.176 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stdout:20 Feb 16:27:16 ntpd[9414]: ntpd: time set -0.061061 s 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stdout:ntpd: time set -0.061061s 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-20T16:27:16.206 INFO:teuthology.orchestra.run.trial013.stderr:20 Feb 16:27:16 ntpd[9414]: can't open /var/log/ntpstats/loopstats.20260220: Permission denied 2026-02-20T16:27:16.270 INFO:teuthology.orchestra.run.trial013.stdout: remote refid st t when poll reach delay offset jitter 2026-02-20T16:27:16.271 INFO:teuthology.orchestra.run.trial013.stdout:============================================================================== 2026-02-20T16:27:16.271 INFO:teuthology.orchestra.run.trial013.stdout: s2-b.time.mci1. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T16:27:16.271 INFO:teuthology.orchestra.run.trial013.stdout: t2.time.bf1.yah .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T16:27:16.271 INFO:teuthology.orchestra.run.trial013.stdout: chi3.us.ntp.li .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T16:27:16.271 INFO:teuthology.orchestra.run.trial013.stdout: pool-96-231-54- .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T16:27:16.272 INFO:teuthology.run_tasks:Running task install... 2026-02-20T16:27:16.286 DEBUG:teuthology.task.install:project ceph 2026-02-20T16:27:16.286 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}} 2026-02-20T16:27:16.286 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'} 2026-02-20T16:27:16.287 INFO:teuthology.task.install:Using flavor: default 2026-02-20T16:27:16.296 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T16:27:16.296 INFO:teuthology.task.install:extra packages: [] 2026-02-20T16:27:16.297 DEBUG:teuthology.orchestra.run.trial013:> sudo apt-key list | grep Ceph 2026-02-20T16:27:16.366 INFO:teuthology.orchestra.run.trial013.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-20T16:27:16.396 INFO:teuthology.orchestra.run.trial013.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-20T16:27:16.396 INFO:teuthology.orchestra.run.trial013.stdout:uid [ unknown] Ceph.com (release key) 2026-02-20T16:27:16.397 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-20T16:27:16.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T16:27:16.566 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default/ 2026-02-20T16:27:16.566 INFO:teuthology.task.install.deb:Package version is 20.2.0-677-gf7870454-1jammy 2026-02-20T16:27:16.715 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:27:16.715 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-20T16:27:16.724 DEBUG:teuthology.orchestra.run.trial013:> sudo apt-get update 2026-02-20T16:27:16.904 INFO:teuthology.orchestra.run.trial013.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T16:27:16.909 INFO:teuthology.orchestra.run.trial013.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-20T16:27:16.918 INFO:teuthology.orchestra.run.trial013.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-20T16:27:16.928 INFO:teuthology.orchestra.run.trial013.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-20T16:27:17.199 INFO:teuthology.orchestra.run.trial013.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy InRelease 2026-02-20T16:27:17.261 INFO:teuthology.orchestra.run.trial013.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-20T16:27:17.323 INFO:teuthology.orchestra.run.trial013.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-20T16:27:17.387 INFO:teuthology.orchestra.run.trial013.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.9 kB] 2026-02-20T16:27:17.539 INFO:teuthology.orchestra.run.trial013.stdout:Fetched 26.5 kB in 1s (40.4 kB/s) 2026-02-20T16:27:18.073 INFO:teuthology.orchestra.run.trial013.stdout:Reading package lists... 2026-02-20T16:27:18.086 DEBUG:teuthology.orchestra.run.trial013:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-677-gf7870454-1jammy cephadm=20.2.0-677-gf7870454-1jammy ceph-mds=20.2.0-677-gf7870454-1jammy ceph-mgr=20.2.0-677-gf7870454-1jammy ceph-common=20.2.0-677-gf7870454-1jammy ceph-fuse=20.2.0-677-gf7870454-1jammy ceph-test=20.2.0-677-gf7870454-1jammy ceph-volume=20.2.0-677-gf7870454-1jammy radosgw=20.2.0-677-gf7870454-1jammy python3-rados=20.2.0-677-gf7870454-1jammy python3-rgw=20.2.0-677-gf7870454-1jammy python3-cephfs=20.2.0-677-gf7870454-1jammy python3-rbd=20.2.0-677-gf7870454-1jammy libcephfs2=20.2.0-677-gf7870454-1jammy libcephfs-dev=20.2.0-677-gf7870454-1jammy librados2=20.2.0-677-gf7870454-1jammy librbd1=20.2.0-677-gf7870454-1jammy rbd-fuse=20.2.0-677-gf7870454-1jammy 2026-02-20T16:27:18.119 INFO:teuthology.orchestra.run.trial013.stdout:Reading package lists... 2026-02-20T16:27:18.229 INFO:teuthology.orchestra.run.trial013.stdout:Building dependency tree... 2026-02-20T16:27:18.230 INFO:teuthology.orchestra.run.trial013.stdout:Reading state information... 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: sg3-utils-udev 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout:The following additional packages will be installed: 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-20T16:27:18.319 INFO:teuthology.orchestra.run.trial013.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: socat xmlstarlet 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout:Suggested packages: 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout:Recommended packages: 2026-02-20T16:27:18.320 INFO:teuthology.orchestra.run.trial013.stdout: btrfs-tools 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout:The following NEW packages will be installed: 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-20T16:27:18.353 INFO:teuthology.orchestra.run.trial013.stdout: socat xmlstarlet 2026-02-20T16:27:18.482 INFO:teuthology.orchestra.run.trial013.stdout:0 upgraded, 86 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T16:27:18.483 INFO:teuthology.orchestra.run.trial013.stdout:Need to get 280 MB of archives. 2026-02-20T16:27:18.483 INFO:teuthology.orchestra.run.trial013.stdout:After this operation, 1119 MB of additional disk space will be used. 2026-02-20T16:27:18.483 INFO:teuthology.orchestra.run.trial013.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-677-gf7870454-1jammy [3562 kB] 2026-02-20T16:27:18.706 INFO:teuthology.orchestra.run.trial013.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-20T16:27:18.988 INFO:teuthology.orchestra.run.trial013.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-677-gf7870454-1jammy [2867 kB] 2026-02-20T16:27:19.043 INFO:teuthology.orchestra.run.trial013.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-20T16:27:19.053 INFO:teuthology.orchestra.run.trial013.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-20T16:27:19.123 INFO:teuthology.orchestra.run.trial013.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-20T16:27:19.324 INFO:teuthology.orchestra.run.trial013.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-20T16:27:19.334 INFO:teuthology.orchestra.run.trial013.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-20T16:27:19.337 INFO:teuthology.orchestra.run.trial013.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-677-gf7870454-1jammy [833 kB] 2026-02-20T16:27:19.360 INFO:teuthology.orchestra.run.trial013.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-20T16:27:19.369 INFO:teuthology.orchestra.run.trial013.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-20T16:27:19.370 INFO:teuthology.orchestra.run.trial013.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-20T16:27:19.371 INFO:teuthology.orchestra.run.trial013.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-20T16:27:19.372 INFO:teuthology.orchestra.run.trial013.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-20T16:27:19.377 INFO:teuthology.orchestra.run.trial013.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-677-gf7870454-1jammy [364 kB] 2026-02-20T16:27:19.389 INFO:teuthology.orchestra.run.trial013.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-20T16:27:19.390 INFO:teuthology.orchestra.run.trial013.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-20T16:27:19.395 INFO:teuthology.orchestra.run.trial013.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-677-gf7870454-1jammy [32.8 kB] 2026-02-20T16:27:19.395 INFO:teuthology.orchestra.run.trial013.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-677-gf7870454-1jammy [184 kB] 2026-02-20T16:27:19.396 INFO:teuthology.orchestra.run.trial013.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-20T16:27:19.400 INFO:teuthology.orchestra.run.trial013.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-677-gf7870454-1jammy [83.9 kB] 2026-02-20T16:27:19.404 INFO:teuthology.orchestra.run.trial013.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-677-gf7870454-1jammy [341 kB] 2026-02-20T16:27:19.421 INFO:teuthology.orchestra.run.trial013.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-677-gf7870454-1jammy [8698 kB] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-20T16:27:19.463 INFO:teuthology.orchestra.run.trial013.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-20T16:27:19.464 INFO:teuthology.orchestra.run.trial013.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-20T16:27:19.467 INFO:teuthology.orchestra.run.trial013.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-20T16:27:19.468 INFO:teuthology.orchestra.run.trial013.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-20T16:27:19.538 INFO:teuthology.orchestra.run.trial013.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-20T16:27:19.540 INFO:teuthology.orchestra.run.trial013.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-20T16:27:19.540 INFO:teuthology.orchestra.run.trial013.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-20T16:27:19.541 INFO:teuthology.orchestra.run.trial013.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-20T16:27:19.566 INFO:teuthology.orchestra.run.trial013.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-20T16:27:19.568 INFO:teuthology.orchestra.run.trial013.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-20T16:27:19.568 INFO:teuthology.orchestra.run.trial013.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-20T16:27:19.626 INFO:teuthology.orchestra.run.trial013.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-20T16:27:19.626 INFO:teuthology.orchestra.run.trial013.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-20T16:27:19.626 INFO:teuthology.orchestra.run.trial013.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-20T16:27:19.627 INFO:teuthology.orchestra.run.trial013.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-20T16:27:19.627 INFO:teuthology.orchestra.run.trial013.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-20T16:27:19.685 INFO:teuthology.orchestra.run.trial013.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-20T16:27:19.687 INFO:teuthology.orchestra.run.trial013.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-20T16:27:19.689 INFO:teuthology.orchestra.run.trial013.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-20T16:27:19.690 INFO:teuthology.orchestra.run.trial013.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-20T16:27:19.732 INFO:teuthology.orchestra.run.trial013.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-677-gf7870454-1jammy [112 kB] 2026-02-20T16:27:19.733 INFO:teuthology.orchestra.run.trial013.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-677-gf7870454-1jammy [261 kB] 2026-02-20T16:27:19.745 INFO:teuthology.orchestra.run.trial013.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-677-gf7870454-1jammy [29.3 MB] 2026-02-20T16:27:19.757 INFO:teuthology.orchestra.run.trial013.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-20T16:27:19.759 INFO:teuthology.orchestra.run.trial013.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-20T16:27:19.761 INFO:teuthology.orchestra.run.trial013.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-20T16:27:19.765 INFO:teuthology.orchestra.run.trial013.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-20T16:27:19.768 INFO:teuthology.orchestra.run.trial013.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-20T16:27:19.768 INFO:teuthology.orchestra.run.trial013.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-20T16:27:19.828 INFO:teuthology.orchestra.run.trial013.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-20T16:27:19.829 INFO:teuthology.orchestra.run.trial013.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-20T16:27:19.830 INFO:teuthology.orchestra.run.trial013.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-20T16:27:19.836 INFO:teuthology.orchestra.run.trial013.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-20T16:27:19.900 INFO:teuthology.orchestra.run.trial013.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-20T16:27:19.902 INFO:teuthology.orchestra.run.trial013.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-20T16:27:19.902 INFO:teuthology.orchestra.run.trial013.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-20T16:27:19.903 INFO:teuthology.orchestra.run.trial013.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-20T16:27:20.545 INFO:teuthology.orchestra.run.trial013.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-677-gf7870454-1jammy [5416 kB] 2026-02-20T16:27:20.670 INFO:teuthology.orchestra.run.trial013.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-677-gf7870454-1jammy [246 kB] 2026-02-20T16:27:20.675 INFO:teuthology.orchestra.run.trial013.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-677-gf7870454-1jammy [124 kB] 2026-02-20T16:27:20.678 INFO:teuthology.orchestra.run.trial013.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-677-gf7870454-1jammy [906 kB] 2026-02-20T16:27:20.700 INFO:teuthology.orchestra.run.trial013.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-677-gf7870454-1jammy [6401 kB] 2026-02-20T16:27:20.849 INFO:teuthology.orchestra.run.trial013.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-677-gf7870454-1jammy [21.7 MB] 2026-02-20T16:27:21.243 INFO:teuthology.orchestra.run.trial013.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-677-gf7870454-1jammy [14.1 kB] 2026-02-20T16:27:21.243 INFO:teuthology.orchestra.run.trial013.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-677-gf7870454-1jammy [957 kB] 2026-02-20T16:27:21.258 INFO:teuthology.orchestra.run.trial013.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-677-gf7870454-1jammy [2345 kB] 2026-02-20T16:27:21.299 INFO:teuthology.orchestra.run.trial013.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-677-gf7870454-1jammy [1049 kB] 2026-02-20T16:27:21.317 INFO:teuthology.orchestra.run.trial013.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-677-gf7870454-1jammy [179 kB] 2026-02-20T16:27:21.320 INFO:teuthology.orchestra.run.trial013.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-677-gf7870454-1jammy [45.5 MB] 2026-02-20T16:27:22.139 INFO:teuthology.orchestra.run.trial013.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-677-gf7870454-1jammy [8625 kB] 2026-02-20T16:27:22.285 INFO:teuthology.orchestra.run.trial013.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-677-gf7870454-1jammy [14.2 kB] 2026-02-20T16:27:22.286 INFO:teuthology.orchestra.run.trial013.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-677-gf7870454-1jammy [99.5 MB] 2026-02-20T16:27:24.076 INFO:teuthology.orchestra.run.trial013.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-677-gf7870454-1jammy [135 kB] 2026-02-20T16:27:24.077 INFO:teuthology.orchestra.run.trial013.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-677-gf7870454-1jammy [43.3 kB] 2026-02-20T16:27:24.077 INFO:teuthology.orchestra.run.trial013.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-677-gf7870454-1jammy [30.7 kB] 2026-02-20T16:27:24.077 INFO:teuthology.orchestra.run.trial013.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-677-gf7870454-1jammy [41.5 kB] 2026-02-20T16:27:24.078 INFO:teuthology.orchestra.run.trial013.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-677-gf7870454-1jammy [25.1 MB] 2026-02-20T16:27:24.515 INFO:teuthology.orchestra.run.trial013.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-677-gf7870454-1jammy [97.9 kB] 2026-02-20T16:27:24.794 INFO:teuthology.orchestra.run.trial013.stdout:Fetched 280 MB in 6s (45.5 MB/s) 2026-02-20T16:27:24.816 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-20T16:27:24.974 INFO:teuthology.orchestra.run.trial013.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-20T16:27:24.976 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-20T16:27:24.977 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T16:27:24.995 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-20T16:27:25.004 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-20T16:27:25.005 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T16:27:25.021 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-20T16:27:25.033 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-20T16:27:25.033 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T16:27:25.052 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-20T16:27:25.064 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T16:27:25.069 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:25.100 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-20T16:27:25.109 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T16:27:25.109 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:25.125 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-20T16:27:25.133 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T16:27:25.133 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:25.153 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-20T16:27:25.161 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-20T16:27:25.161 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T16:27:25.177 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package librados2. 2026-02-20T16:27:25.185 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../07-librados2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.185 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.226 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libnbd0. 2026-02-20T16:27:25.234 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-20T16:27:25.235 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-20T16:27:25.249 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package librbd1. 2026-02-20T16:27:25.257 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../09-librbd1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.257 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.295 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libcephfs2. 2026-02-20T16:27:25.303 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.303 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.323 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-rados. 2026-02-20T16:27:25.331 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../11-python3-rados_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.331 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.348 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-20T16:27:25.357 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:25.357 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.371 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-cephfs. 2026-02-20T16:27:25.379 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.379 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.394 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-20T16:27:25.402 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:25.403 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.421 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-20T16:27:25.429 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-20T16:27:25.429 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T16:27:25.445 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-prettytable. 2026-02-20T16:27:25.453 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-20T16:27:25.454 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-20T16:27:25.468 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-rbd. 2026-02-20T16:27:25.476 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.476 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.493 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-20T16:27:25.501 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-20T16:27:25.501 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T16:27:25.519 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package librgw2. 2026-02-20T16:27:25.527 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../19-librgw2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.527 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.643 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-rgw. 2026-02-20T16:27:25.651 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.652 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.667 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-20T16:27:25.675 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-20T16:27:25.675 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T16:27:25.690 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libradosstriper1. 2026-02-20T16:27:25.698 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.699 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.716 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-common. 2026-02-20T16:27:25.724 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../23-ceph-common_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.724 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:25.983 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-base. 2026-02-20T16:27:25.992 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../24-ceph-base_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:25.996 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.065 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-20T16:27:26.074 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-20T16:27:26.074 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-20T16:27:26.089 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-cheroot. 2026-02-20T16:27:26.097 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-20T16:27:26.097 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T16:27:26.114 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-20T16:27:26.122 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-20T16:27:26.122 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-20T16:27:26.136 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-20T16:27:26.144 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-20T16:27:26.144 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-20T16:27:26.158 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-20T16:27:26.166 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-20T16:27:26.166 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-20T16:27:26.180 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-tempora. 2026-02-20T16:27:26.188 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-20T16:27:26.189 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-20T16:27:26.203 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-portend. 2026-02-20T16:27:26.211 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-20T16:27:26.211 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-20T16:27:26.226 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-20T16:27:26.234 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-20T16:27:26.234 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-20T16:27:26.249 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-20T16:27:26.257 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-20T16:27:26.257 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-20T16:27:26.280 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-natsort. 2026-02-20T16:27:26.288 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-20T16:27:26.289 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-20T16:27:26.304 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-20T16:27:26.312 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:26.312 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.339 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-20T16:27:26.347 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.347 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.362 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr. 2026-02-20T16:27:26.370 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.370 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.390 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mon. 2026-02-20T16:27:26.398 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.399 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.457 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-20T16:27:26.465 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-20T16:27:26.465 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T16:27:26.481 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-osd. 2026-02-20T16:27:26.489 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.489 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.635 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph. 2026-02-20T16:27:26.644 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../41-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.644 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.658 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-fuse. 2026-02-20T16:27:26.666 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.666 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.688 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mds. 2026-02-20T16:27:26.696 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.697 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.728 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package cephadm. 2026-02-20T16:27:26.736 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../44-cephadm_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:26.736 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.753 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-20T16:27:26.761 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T16:27:26.761 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T16:27:26.782 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-20T16:27:26.790 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:26.790 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:26.811 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-20T16:27:26.819 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-20T16:27:26.820 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-20T16:27:26.835 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-routes. 2026-02-20T16:27:26.843 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-20T16:27:26.843 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T16:27:26.863 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-20T16:27:26.871 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:26.871 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:27.306 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-20T16:27:27.316 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-20T16:27:27.316 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T16:27:27.357 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-joblib. 2026-02-20T16:27:27.365 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-20T16:27:27.365 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T16:27:27.396 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-20T16:27:27.404 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-20T16:27:27.405 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-20T16:27:27.419 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-sklearn. 2026-02-20T16:27:27.428 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-20T16:27:27.428 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T16:27:27.512 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-20T16:27:27.521 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:27.521 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:27.686 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-cachetools. 2026-02-20T16:27:27.695 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-20T16:27:27.695 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-20T16:27:27.710 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-rsa. 2026-02-20T16:27:27.718 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-20T16:27:27.719 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-20T16:27:27.737 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-google-auth. 2026-02-20T16:27:27.745 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-20T16:27:27.745 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-20T16:27:27.763 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-20T16:27:27.771 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-20T16:27:27.771 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T16:27:27.787 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-websocket. 2026-02-20T16:27:27.796 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-20T16:27:27.796 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-20T16:27:27.813 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-20T16:27:27.821 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-20T16:27:27.822 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T16:27:27.914 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-20T16:27:27.923 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:27.923 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:27.938 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-20T16:27:27.946 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-20T16:27:27.946 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T16:27:27.962 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-20T16:27:27.970 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T16:27:27.970 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T16:27:27.985 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package jq. 2026-02-20T16:27:27.993 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T16:27:27.993 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-20T16:27:28.008 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package socat. 2026-02-20T16:27:28.016 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-20T16:27:28.016 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-20T16:27:28.036 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package xmlstarlet. 2026-02-20T16:27:28.044 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-20T16:27:28.045 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-20T16:27:28.081 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-test. 2026-02-20T16:27:28.089 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../67-ceph-test_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:28.090 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.078 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package ceph-volume. 2026-02-20T16:27:29.087 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T16:27:29.087 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.110 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-20T16:27:29.119 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:29.119 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.134 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-20T16:27:29.143 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:29.143 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.158 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-20T16:27:29.167 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:29.167 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.183 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package nvme-cli. 2026-02-20T16:27:29.191 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-20T16:27:29.191 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T16:27:29.224 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-20T16:27:29.233 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T16:27:29.233 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T16:27:29.262 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-20T16:27:29.271 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-20T16:27:29.271 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-20T16:27:29.285 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-pluggy. 2026-02-20T16:27:29.294 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-20T16:27:29.294 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-20T16:27:29.310 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-psutil. 2026-02-20T16:27:29.319 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-20T16:27:29.319 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-20T16:27:29.337 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-py. 2026-02-20T16:27:29.345 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-20T16:27:29.346 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-20T16:27:29.367 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-pygments. 2026-02-20T16:27:29.375 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-20T16:27:29.376 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T16:27:29.418 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-toml. 2026-02-20T16:27:29.427 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-20T16:27:29.427 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-20T16:27:29.442 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-pytest. 2026-02-20T16:27:29.451 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-20T16:27:29.451 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T16:27:29.474 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-simplejson. 2026-02-20T16:27:29.482 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-20T16:27:29.483 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-20T16:27:29.500 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package python3-webob. 2026-02-20T16:27:29.509 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-20T16:27:29.509 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T16:27:29.532 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-20T16:27:29.540 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-20T16:27:29.541 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-20T16:27:29.608 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package radosgw. 2026-02-20T16:27:29.617 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../84-radosgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:29.617 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.828 INFO:teuthology.orchestra.run.trial013.stdout:Selecting previously unselected package rbd-fuse. 2026-02-20T16:27:29.837 INFO:teuthology.orchestra.run.trial013.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T16:27:29.838 INFO:teuthology.orchestra.run.trial013.stdout:Unpacking rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:29.864 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-20T16:27:29.932 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T16:27:29.932 INFO:teuthology.orchestra.run.trial013.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T16:27:30.003 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T16:27:30.276 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-20T16:27:30.639 INFO:teuthology.orchestra.run.trial013.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-20T16:27:30.651 INFO:teuthology.orchestra.run.trial013.stdout:Setting up cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:30.695 INFO:teuthology.orchestra.run.trial013.stdout:Adding system user cephadm....done 2026-02-20T16:27:30.707 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-20T16:27:30.772 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T16:27:30.773 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-20T16:27:30.844 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-20T16:27:30.917 INFO:teuthology.orchestra.run.trial013.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T16:27:30.918 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-20T16:27:31.015 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T16:27:31.138 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-20T16:27:31.214 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-20T16:27:31.289 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:31.353 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T16:27:31.354 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-20T16:27:31.355 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T16:27:31.356 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T16:27:31.357 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-20T16:27:31.479 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-20T16:27:31.548 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:31.548 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T16:27:31.625 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-20T16:27:31.711 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T16:27:31.961 INFO:teuthology.orchestra.run.trial013.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-20T16:27:31.962 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T16:27:32.054 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T16:27:32.180 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T16:27:32.261 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-20T16:27:32.332 INFO:teuthology.orchestra.run.trial013.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-20T16:27:32.333 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:32.420 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T16:27:32.913 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:32.918 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-20T16:27:32.992 INFO:teuthology.orchestra.run.trial013.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T16:27:32.993 INFO:teuthology.orchestra.run.trial013.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-20T16:27:32.994 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-20T16:27:33.071 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-20T16:27:33.143 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:33.144 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-20T16:27:33.221 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-20T16:27:33.294 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-20T16:27:33.371 INFO:teuthology.orchestra.run.trial013.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T16:27:33.372 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-20T16:27:33.454 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T16:27:33.455 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T16:27:33.531 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T16:27:33.615 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-20T16:27:33.689 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T16:27:33.690 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T16:27:33.821 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-20T16:27:33.892 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T16:27:33.893 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-20T16:27:33.973 INFO:teuthology.orchestra.run.trial013.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-20T16:27:33.974 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-20T16:27:34.102 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T16:27:34.103 INFO:teuthology.orchestra.run.trial013.stdout:Setting up librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.104 INFO:teuthology.orchestra.run.trial013.stdout:Setting up librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.105 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.106 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T16:27:34.621 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.622 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.623 INFO:teuthology.orchestra.run.trial013.stdout:Setting up librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.624 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.624 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:34.696 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T16:27:34.696 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T16:27:35.077 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.078 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.079 INFO:teuthology.orchestra.run.trial013.stdout:Setting up libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.080 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.081 INFO:teuthology.orchestra.run.trial013.stdout:Setting up rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.082 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.082 INFO:teuthology.orchestra.run.trial013.stdout:Setting up python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.083 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.111 INFO:teuthology.orchestra.run.trial013.stdout:Adding group ceph....done 2026-02-20T16:27:35.148 INFO:teuthology.orchestra.run.trial013.stdout:Adding system user ceph....done 2026-02-20T16:27:35.157 INFO:teuthology.orchestra.run.trial013.stdout:Setting system user ceph properties....done 2026-02-20T16:27:35.164 INFO:teuthology.orchestra.run.trial013.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-20T16:27:35.250 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-20T16:27:35.494 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-20T16:27:35.846 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:35.847 INFO:teuthology.orchestra.run.trial013.stdout:Setting up radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:36.086 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T16:27:36.086 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T16:27:36.446 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:36.564 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-20T16:27:36.943 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:37.002 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T16:27:37.002 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T16:27:37.375 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:37.446 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T16:27:37.446 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T16:27:37.829 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:37.925 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T16:27:37.926 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T16:27:38.310 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.319 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.329 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.399 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T16:27:38.400 INFO:teuthology.orchestra.run.trial013.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T16:27:38.773 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.783 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.784 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.795 INFO:teuthology.orchestra.run.trial013.stdout:Setting up ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T16:27:38.904 INFO:teuthology.orchestra.run.trial013.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-20T16:27:38.987 INFO:teuthology.orchestra.run.trial013.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-20T16:27:39.399 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.399 INFO:teuthology.orchestra.run.trial013.stdout:Running kernel seems to be up-to-date. 2026-02-20T16:27:39.399 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.399 INFO:teuthology.orchestra.run.trial013.stdout:The processor microcode seems to be up-to-date. 2026-02-20T16:27:39.399 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout:No services need to be restarted. 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout:No containers need to be restarted. 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout:No user sessions are running outdated binaries. 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout: 2026-02-20T16:27:39.400 INFO:teuthology.orchestra.run.trial013.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-20T16:27:40.208 INFO:teuthology.orchestra.run.trial013.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-20T16:27:40.215 DEBUG:teuthology.parallel:result is None 2026-02-20T16:27:40.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T16:27:40.386 DEBUG:teuthology.orchestra.run.trial013:> dpkg-query -W -f '${Version}' ceph 2026-02-20T16:27:40.396 INFO:teuthology.orchestra.run.trial013.stdout:20.2.0-677-gf7870454-1jammy 2026-02-20T16:27:40.397 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-677-gf7870454-1jammy 2026-02-20T16:27:40.397 INFO:teuthology.task.install:The correct ceph version 20.2.0-677-gf7870454-1jammy is installed. 2026-02-20T16:27:40.398 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-20T16:27:40.398 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:27:40.398 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T16:27:40.447 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-20T16:27:40.447 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:27:40.447 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T16:27:40.499 DEBUG:teuthology.orchestra.run.trial013:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T16:27:40.551 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-20T16:27:40.551 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:27:40.552 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T16:27:40.603 DEBUG:teuthology.orchestra.run.trial013:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T16:27:40.655 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-20T16:27:40.655 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:27:40.655 DEBUG:teuthology.orchestra.run.trial013:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T16:27:40.703 DEBUG:teuthology.orchestra.run.trial013:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T16:27:40.751 INFO:teuthology.run_tasks:Running task workunit... 2026-02-20T16:27:40.779 INFO:tasks.workunit:Pulling workunits from ref f7870454149f22574086dd2ff55d050585136531 2026-02-20T16:27:40.780 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-20T16:27:40.780 INFO:tasks.workunit:timeout=3h 2026-02-20T16:27:40.780 INFO:tasks.workunit:cleanup=True 2026-02-20T16:27:40.780 DEBUG:teuthology.orchestra.run.trial013:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-20T16:27:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T16:27:40.795 INFO:teuthology.orchestra.run.trial013.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-20T16:27:40.795 DEBUG:teuthology.orchestra.run.trial013:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-20T16:27:40.843 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-20T16:27:40.843 DEBUG:teuthology.orchestra.run.trial013:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-20T16:27:40.891 DEBUG:teuthology.orchestra.run.trial013:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f7870454149f22574086dd2ff55d050585136531 2026-02-20T16:27:40.942 INFO:tasks.workunit.client.0.trial013.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-20T16:28:02.947 INFO:tasks.workunit.client.0.trial013.stderr:Note: switching to 'f7870454149f22574086dd2ff55d050585136531'. 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr:state without impacting any branches by switching back to a branch. 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.948 INFO:tasks.workunit.client.0.trial013.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: git switch -c 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr:Or undo this operation with: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: git switch - 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr: 2026-02-20T16:28:02.949 INFO:tasks.workunit.client.0.trial013.stderr:HEAD is now at f7870454149 ceph-dev-pipeline: configure 2026-02-20T16:28:02.958 DEBUG:teuthology.orchestra.run.trial013:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-20T16:28:03.004 DEBUG:teuthology.orchestra.run.trial013:> set -ex 2026-02-20T16:28:03.005 DEBUG:teuthology.orchestra.run.trial013:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-20T16:28:03.053 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2026-02-20T16:28:03.054 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2026-02-20T16:28:03.055 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2026-02-20T16:28:03.105 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T16:28:03.110 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T16:28:03.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:28:03.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-mapper 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2026-02-20T16:28:03.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2026-02-20T16:28:03.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2026-02-20T16:28:03.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none ' 2026-02-20T16:28:03.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-20T16:28:03.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T16:28:03.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2026-02-20T16:28:03.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T16:28:03.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2026-02-20T16:28:03.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:03.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:03.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:03.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:28:03.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:28:03.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:28:03.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:28:03.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:28:03.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:28:03.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:03.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:28:03.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:28:03.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:03.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:28:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:28:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-20T16:28:03.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:28:03.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:03.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:03.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12019 2026-02-20T16:28:03.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:28:03.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:28:03.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2026-02-20T16:28:03.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:28:03.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:03.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:03.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12019 2026-02-20T16:28:03.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:28:03.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T16:28:03.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2026-02-20T16:28:03.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2026-02-20T16:28:03.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2026-02-20T16:28:03.134 INFO:tasks.workunit.client.0.trial013.stdout:Dir: td/osd-mapper 2026-02-20T16:28:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-20T16:28:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-20T16:28:03.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-20T16:28:03.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-20T16:28:03.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T16:28:03.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T16:28:03.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T16:28:03.427 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 c31287ff-3fd4-4416-a367-ef0b71d0cbf8 2026-02-20T16:28:03.524 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:03.552 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:03.548+0000 7f01e8231a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:03.553 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:03.548+0000 7f01e8231a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:03.554 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:03.548+0000 7f01e8231a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:03.555 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:03.548+0000 7f01e8231a40 -1 bdev(0x55877205ac00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:28:03.555 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:03.548+0000 7f01e8231a40 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2026-02-20T16:28:04.244 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:28:04.338 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:28:04.355 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:04.352+0000 7fdd66eeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:04.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:04.352+0000 7fdd66eeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:04.358 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:04.352+0000 7fdd66eeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:04.429 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:04.744 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:04.740+0000 7fdd66eeda40 -1 Falling back to public interface 2026-02-20T16:28:05.520 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:05.774 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:05.768+0000 7fdd66eeda40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:28:06.612 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:28:06.772 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3947611515,v1:127.0.0.1:6801/3947611515] [v2:127.0.0.1:6802/3947611515,v1:127.0.0.1:6803/3947611515] exists,up c31287ff-3fd4-4416-a367-ef0b71d0cbf8 2026-02-20T16:28:06.777 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 1af5d89d-168e-4f6d-a358-0967676f6a48 2026-02-20T16:28:06.951 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:06.987 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:06.984+0000 7fe51f580a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:06.989 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:06.984+0000 7fe51f580a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:06.990 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:06.984+0000 7fe51f580a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:06.991 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:06.984+0000 7fe51f580a40 -1 bdev(0x55a925e2ec00 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:28:06.991 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:06.988+0000 7fe51f580a40 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2026-02-20T16:28:07.639 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:28:07.839 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:28:07.868 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:07.864+0000 7f3ede536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:07.869 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:07.864+0000 7f3ede536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:07.870 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:07.864+0000 7f3ede536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:08.001 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:08.293 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:08.288+0000 7f3ede536a40 -1 Falling back to public interface 2026-02-20T16:28:08.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:08.840+0000 7f3ede536a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:28:09.159 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:10.318 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:28:10.480 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2431144345,v1:127.0.0.1:6811/2431144345] [v2:127.0.0.1:6812/2431144345,v1:127.0.0.1:6813/2431144345] exists,up 1af5d89d-168e-4f6d-a358-0967676f6a48 2026-02-20T16:28:10.486 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 697067cf-a78a-4a0e-a70c-a8be39ea62af 2026-02-20T16:28:10.659 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:28:10.688 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:10.684+0000 7f5c78dfba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:10.689 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:10.684+0000 7f5c78dfba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:10.690 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:10.684+0000 7f5c78dfba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:10.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:10.688+0000 7f5c78dfba40 -1 bdev(0x56085bc20c00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:28:10.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:10.688+0000 7f5c78dfba40 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2026-02-20T16:28:11.323 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:28:11.528 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:28:11.557 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:11.552+0000 7f7b10ad5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:11.558 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:11.552+0000 7f7b10ad5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:11.559 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:11.556+0000 7f7b10ad5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:11.688 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:12.032 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:12.028+0000 7f7b10ad5a40 -1 Falling back to public interface 2026-02-20T16:28:12.702 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:12.696+0000 7f7b10ad5a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:28:12.849 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:14.015 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:28:14.173 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2050177717,v1:127.0.0.1:6819/2050177717] [v2:127.0.0.1:6820/2050177717,v1:127.0.0.1:6821/2050177717] exists,up 697067cf-a78a-4a0e-a70c-a8be39ea62af 2026-02-20T16:28:14.354 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T16:28:15.837 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T16:28:17.163 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705671 2026-02-20T16:28:17.326 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574852 2026-02-20T16:28:19.194 INFO:tasks.workunit.client.0.trial013.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-20T16:28:19.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-20T16:28:19.256 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T16:28:19.256 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T16:28:19.256 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.263 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T16:28:19.264 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T16:28:19.264 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.270 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T16:28:19.271 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T16:28:19.271 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2026-02-20T16:28:19.342 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T16:28:19.342 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-20T16:28:19.342 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.349 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T16:28:19.349 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-20T16:28:19.349 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.356 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T16:28:19.356 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-20T16:28:19.356 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-20T16:28:19.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-20T16:28:19.437 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap01 2026-02-20T16:28:19.539 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap02 2026-02-20T16:28:19.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2026-02-20T16:28:19.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-20T16:28:19.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-20T16:28:19.641 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap13 2026-02-20T16:28:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2026-02-20T16:28:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-20T16:28:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-20T16:28:19.744 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap24 2026-02-20T16:28:19.846 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap25 2026-02-20T16:28:19.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 12146 2026-02-20T16:28:19.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2026-02-20T16:28:19.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-20T16:28:19.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2026-02-20T16:28:20.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2026-02-20T16:28:20.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-20T16:28:20.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-20T16:28:20.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2026-02-20T16:28:20.187 INFO:tasks.workunit.client.0.trial013.stdout:pgid is 1.3 (primary: osd.1) 2026-02-20T16:28:20.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2026-02-20T16:28:20.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2026-02-20T16:28:20.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.3 2026-02-20T16:28:20.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-20T16:28:20.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-20T16:28:20.336 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs_brief 2026-02-20T16:28:20.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-20T16:28:20.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2026-02-20T16:28:20.346 INFO:tasks.workunit.client.0.trial013.stdout:Setting scrub debug data. Primary for 1.3 is 1 2026-02-20T16:28:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-20T16:28:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:28:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12019/ceph-osd.1.asok 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-20T16:28:20.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.12019/ceph-osd.1.asok scrubdebug 1.3 set sessions 2026-02-20T16:28:20.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:20.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout:{"success":true}{ 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "name": "objxxx", 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 5, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "id": 2, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "id": 1, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap01" 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "id": 2, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap02" 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6, 2026-02-20T16:28:20.432 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": 3, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": 3, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap13" 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": 5, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": 4, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap24" 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": 5, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap25" 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "size": 5, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:20.433 INFO:tasks.workunit.client.0.trial013.stdout: "id": "head", 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [], 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:20.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-20T16:28:20.499 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:20.500 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:28:20.500 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T16:28:20.500 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T16:28:20.500 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:20.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2026-02-20T16:28:23.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-20T16:28:23.652 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-20T16:28:20.800288+0000 23'7 23:42 [1,2,0] 1 [1,2,0] 1 23'7 2026-02-20T16:28:20.800226+0000 23'7 2026-02-20T16:28:20.800226+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:28:20.800226+0000 4 0 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.871741+0000 0'0 23:29 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.650565+0000 0'0 23:32 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.650335+0000 0'0 23:30 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:23.653 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:28:23.654 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:28:23.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2026-02-20T16:28:23.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-20T16:28:20.800288+0000 23'7 23:42 [1,2,0] 1 [1,2,0] 1 23'7 2026-02-20T16:28:20.800226+0000 23'7 2026-02-20T16:28:20.800226+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:28:20.800226+0000 4 0 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.871741+0000 0'0 23:29 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.650565+0000 0'0 23:32 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:14.650335+0000 0'0 23:30 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:23.809 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:23.810 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:28:23.810 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:28:23.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2026-02-20T16:28:24.033 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:28:24.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2026-02-20T16:28:24.238 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:28:24.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2026-02-20T16:28:29.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:29.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-20T16:28:29.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:29.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:29.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:29.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:29.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:29.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:29.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:29.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-20T16:28:29.575 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.572+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:29.575 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.572+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.576 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.572+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) close 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bluefs mount 2026-02-20T16:28:29.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.628+0000 7f8a2463bd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:29.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluefs mount shared_bdev_used = 0 2026-02-20T16:28:29.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.632+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:29.647 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:29.647 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2026-02-20T16:28:29.647 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-20T16:28:29.647 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-20T16:28:29.647 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-20T16:28:29.648 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 freelist init 2026-02-20T16:28:29.648 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 freelist _read_cfg 2026-02-20T16:28:29.648 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-20T16:28:29.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 7.62941e-08 2026-02-20T16:28:29.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bluefs umount 2026-02-20T16:28:29.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.644+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) close 2026-02-20T16:28:29.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:29.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.696 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-20T16:28:29.696 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount 2026-02-20T16:28:29.696 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:29.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.698 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.698 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-20T16:28:29.698 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.692+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:29.709 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.704+0000 7f8a2463bd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000000 0b 05 58 02 00 00 45 f1 f7 30 fe 23 45 e2 82 e9 |..X...E..0.#E...| 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000010 a6 20 74 37 61 96 00 00 00 00 19 00 00 00 00 00 |. t7a...........| 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:* 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-20T16:28:29.710 INFO:tasks.workunit.client.0.trial013.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-20T16:28:29.711 INFO:tasks.workunit.client.0.trial013.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-20T16:28:29.712 INFO:tasks.workunit.client.0.trial013.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000220 72 65 19 00 00 00 05 00 00 00 c3 12 87 ff 3f d4 |re............?.| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000230 44 16 a3 67 ef 0b 71 d0 cb f8 00 00 00 00 00 00 |D..g..q.........| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000240 00 00 19 00 00 00 96 8b 98 69 5e b5 6b 2f 01 00 |.........i^.k/..| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:0000025e 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.713 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-20T16:28:29.714 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:0000003f 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:29.715 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:29.716 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:29.717 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-20T16:28:29.718 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.719 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-20T16:28:29.719 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:29.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.704+0000 7f8a2463bd40 1 bluefs umount 2026-02-20T16:28:29.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.704+0000 7f8a2463bd40 1 bdev(0x55b627ba9c00 td/osd-mapper/0/block) close 2026-02-20T16:28:29.751 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.748+0000 7f8a2463bd40 1 freelist shutdown 2026-02-20T16:28:29.751 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.748+0000 7f8a2463bd40 1 bdev(0x55b627ba9800 td/osd-mapper/0/block) close 2026-02-20T16:28:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:29.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-20T16:28:29.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2026-02-20T16:28:29.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.800+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:29.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.800+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.800+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) close 2026-02-20T16:28:29.863 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:29.863 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-20T16:28:29.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:29.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-20T16:28:29.865 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount 2026-02-20T16:28:29.865 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluefs mount shared_bdev_used = 0 2026-02-20T16:28:29.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.860+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:29.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:29.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-20T16:28:29.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-20T16:28:29.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-20T16:28:29.880 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-20T16:28:29.880 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 freelist init 2026-02-20T16:28:29.880 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 freelist _read_cfg 2026-02-20T16:28:29.880 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-20T16:28:29.881 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc4000, fragmentation 7.62941e-08 2026-02-20T16:28:29.881 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bluefs umount 2026-02-20T16:28:29.881 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.876+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) close 2026-02-20T16:28:29.939 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:29.940 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:29.940 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-20T16:28:29.940 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount 2026-02-20T16:28:29.940 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-20T16:28:29.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.936+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:29.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.948+0000 7fabe9795d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:29.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.948+0000 7fabe9795d40 1 bluefs umount 2026-02-20T16:28:29.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.948+0000 7fabe9795d40 1 bdev(0x55577ba3fc00 td/osd-mapper/2/block) close 2026-02-20T16:28:29.995 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.992+0000 7fabe9795d40 1 freelist shutdown 2026-02-20T16:28:29.995 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:29.992+0000 7fabe9795d40 1 bdev(0x55577ba3f800 td/osd-mapper/2/block) close 2026-02-20T16:28:30.043 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.043 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.043 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:30.043 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.043 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:30.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2026-02-20T16:28:30.044 INFO:tasks.workunit.client.0.trial013.stderr:grep: /tmp/oo2.dump: No such file or directory 2026-02-20T16:28:30.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:30.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-20T16:28:30.197 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-20T16:28:30.197 INFO:tasks.workunit.client.0.trial013.stdout:00000000 0b 05 58 02 00 00 45 f1 f7 30 fe 23 45 e2 82 e9 |..X...E..0.#E...| 2026-02-20T16:28:30.197 INFO:tasks.workunit.client.0.trial013.stdout:00000010 a6 20 74 37 61 96 02 00 00 00 1a 00 00 00 00 00 |. t7a...........| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:* 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-20T16:28:30.198 INFO:tasks.workunit.client.0.trial013.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-20T16:28:30.199 INFO:tasks.workunit.client.0.trial013.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000220 72 65 1a 00 00 00 0d 00 00 00 69 70 67 cf a7 8a |re........ipg...| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000230 4a 0e a7 0c a8 be 39 ea 62 af 00 00 00 00 00 00 |J.....9.b.......| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000240 00 00 1a 00 00 00 9d 8b 98 69 ba f3 d4 2b 01 00 |.........i...+..| 2026-02-20T16:28:30.200 INFO:tasks.workunit.client.0.trial013.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1a 00 00 00 |..............| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:0000025e 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:30.201 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:0000003f 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.202 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.203 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.204 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.205 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-20T16:28:30.206 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-20T16:28:30.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-20T16:28:30.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:30.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2026-02-20T16:28:30.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2026-02-20T16:28:30.560 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2026-02-20T16:28:30.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2026-02-20T16:28:30.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:30.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2026-02-20T16:28:30.563 INFO:tasks.workunit.client.0.trial013.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2026-02-20T16:28:30.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2026-02-20T16:28:30.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-20T16:28:30.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-20T16:28:30.584 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.580+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:30.584 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.580+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:30.584 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.580+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) close 2026-02-20T16:28:30.631 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:30.631 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:30.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-20T16:28:30.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:30.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:30.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-20T16:28:30.633 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bluefs mount 2026-02-20T16:28:30.633 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.628+0000 7fe5cb047d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluefs mount shared_bdev_used = 0 2026-02-20T16:28:30.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.632+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 freelist init 2026-02-20T16:28:30.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 freelist _read_cfg 2026-02-20T16:28:30.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.644+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-20T16:28:30.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.648+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 7.62941e-08 2026-02-20T16:28:30.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.648+0000 7fe5cb047d40 1 bluefs umount 2026-02-20T16:28:30.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.648+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) close 2026-02-20T16:28:30.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-20T16:28:30.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:30.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-20T16:28:30.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount 2026-02-20T16:28:30.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-20T16:28:30.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.708+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:30.724 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.720+0000 7fe5cb047d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000000 0b 05 58 02 00 00 45 f1 f7 30 fe 23 45 e2 82 e9 |..X...E..0.#E...| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000010 a6 20 74 37 61 96 00 00 00 00 19 00 00 00 00 00 |. t7a...........| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:* 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-20T16:28:30.725 INFO:tasks.workunit.client.0.trial013.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-20T16:28:30.726 INFO:tasks.workunit.client.0.trial013.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000220 72 65 19 00 00 00 05 00 00 00 c3 12 87 ff 3f d4 |re............?.| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000230 44 16 a3 67 ef 0b 71 d0 cb f8 00 00 00 00 00 00 |D..g..q.........| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000240 00 00 19 00 00 00 96 8b 98 69 5e b5 6b 2f 01 00 |.........i^.k/..| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:0000025e 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.727 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:0000003f 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:30.728 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.729 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.730 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-20T16:28:30.731 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.731 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-20T16:28:30.731 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:30.731 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.720+0000 7fe5cb047d40 1 bluefs umount 2026-02-20T16:28:30.731 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.720+0000 7fe5cb047d40 1 bdev(0x555a26065c00 td/osd-mapper/0/block) close 2026-02-20T16:28:30.775 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.772+0000 7fe5cb047d40 1 freelist shutdown 2026-02-20T16:28:30.775 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:30.772+0000 7fe5cb047d40 1 bdev(0x555a26065800 td/osd-mapper/0/block) close 2026-02-20T16:28:30.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-20T16:28:30.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-20T16:28:30.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:31.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-20T16:28:31.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-20T16:28:31.048 INFO:tasks.workunit.client.0.trial013.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:31.049 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-20T16:28:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.9v0s9RpjBs_the_val 2026-02-20T16:28:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.9v0s9RpjBs_the_val' 2026-02-20T16:28:31.050 INFO:tasks.workunit.client.0.trial013.stdout:Value dumped in: /tmp/tmp.9v0s9RpjBs_the_val 2026-02-20T16:28:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.9v0s9RpjBs_the_val 2026-02-20T16:28:31.189 INFO:tasks.workunit.client.0.trial013.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-20T16:28:31.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:31.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.9v0s9RpjBs_the_val 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-20T16:28:31.293 INFO:tasks.workunit.client.0.trial013.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-20T16:28:31.294 INFO:tasks.workunit.client.0.trial013.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-20T16:28:31.294 INFO:tasks.workunit.client.0.trial013.stdout:0000073 2026-02-20T16:28:31.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-20T16:28:31.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:31.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.9v0s9RpjBs_the_val 2026-02-20T16:28:31.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:31.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:31.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-20T16:28:31.764 INFO:tasks.workunit.client.0.trial013.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2026-02-20T16:28:31.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2026-02-20T16:28:31.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-20T16:28:31.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-20T16:28:31.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.772+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-20T16:28:31.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.772+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:31.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.772+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) close 2026-02-20T16:28:31.839 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-20T16:28:31.839 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:31.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-20T16:28:31.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-20T16:28:31.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:31.841 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-20T16:28:31.841 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bluefs mount 2026-02-20T16:28:31.841 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.836+0000 7efe15462d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:31.843 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluefs mount shared_bdev_used = 0 2026-02-20T16:28:31.844 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.840+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:31.856 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:31.856 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2051 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 freelist init 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 freelist _read_cfg 2026-02-20T16:28:31.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-20T16:28:31.858 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc4000, fragmentation 7.62941e-08 2026-02-20T16:28:31.858 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bluefs umount 2026-02-20T16:28:31.859 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.852+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) close 2026-02-20T16:28:31.907 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-20T16:28:31.907 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:31.907 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-20T16:28:31.908 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount 2026-02-20T16:28:31.908 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:31.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-20T16:28:31.910 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.904+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:31.923 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.916+0000 7efe15462d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:31.923 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-20T16:28:31.923 INFO:tasks.workunit.client.0.trial013.stdout:00000000 0b 05 58 02 00 00 45 f1 f7 30 fe 23 45 e2 82 e9 |..X...E..0.#E...| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000010 a6 20 74 37 61 96 01 00 00 00 1a 00 00 00 00 00 |. t7a...........| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:* 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-20T16:28:31.924 INFO:tasks.workunit.client.0.trial013.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-20T16:28:31.925 INFO:tasks.workunit.client.0.trial013.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000220 72 65 1a 00 00 00 09 00 00 00 1a f5 d8 9d 16 8e |re..............| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000230 4f 6d a3 58 09 67 67 6f 6a 48 00 00 00 00 00 00 |Om.X.ggojH......| 2026-02-20T16:28:31.926 INFO:tasks.workunit.client.0.trial013.stdout:00000240 00 00 1a 00 00 00 99 8b 98 69 17 dc 95 33 01 00 |.........i...3..| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1a 00 00 00 |..............| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:0000025e 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:31.927 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:0000003f 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:31.928 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.929 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:31.930 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:31.931 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:31.932 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.916+0000 7efe15462d40 1 bluefs umount 2026-02-20T16:28:31.933 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.916+0000 7efe15462d40 1 bdev(0x55bbfb7cdc00 td/osd-mapper/1/block) close 2026-02-20T16:28:31.975 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.972+0000 7efe15462d40 1 freelist shutdown 2026-02-20T16:28:31.975 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:31.972+0000 7efe15462d40 1 bdev(0x55bbfb7cd800 td/osd-mapper/1/block) close 2026-02-20T16:28:32.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-20T16:28:32.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-20T16:28:32.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-20T16:28:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-20T16:28:32.251 INFO:tasks.workunit.client.0.trial013.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:32.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-20T16:28:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.uZ7r8IvPzJ_the_val 2026-02-20T16:28:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:32.253 INFO:tasks.workunit.client.0.trial013.stdout:Value dumped in: /tmp/tmp.uZ7r8IvPzJ_the_val 2026-02-20T16:28:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.uZ7r8IvPzJ_the_val' 2026-02-20T16:28:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.uZ7r8IvPzJ_the_val 2026-02-20T16:28:32.408 INFO:tasks.workunit.client.0.trial013.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-20T16:28:32.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:32.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.uZ7r8IvPzJ_the_val 2026-02-20T16:28:32.554 INFO:tasks.workunit.client.0.trial013.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-20T16:28:32.554 INFO:tasks.workunit.client.0.trial013.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stdout:0000073 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-20T16:28:32.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:32.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.uZ7r8IvPzJ_the_val 2026-02-20T16:28:33.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:33.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:33.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2026-02-20T16:28:33.014 INFO:tasks.workunit.client.0.trial013.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2026-02-20T16:28:33.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2026-02-20T16:28:33.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-20T16:28:33.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-20T16:28:33.025 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.020+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:33.025 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.020+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:33.026 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.020+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) close 2026-02-20T16:28:33.071 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount 2026-02-20T16:28:33.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:33.074 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.074 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluefs mount shared_bdev_used = 0 2026-02-20T16:28:33.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.068+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:33.086 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 freelist init 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 freelist _read_cfg 2026-02-20T16:28:33.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.080+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-20T16:28:33.088 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.084+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc4000, fragmentation 7.62941e-08 2026-02-20T16:28:33.088 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.084+0000 7fbd26c86d40 1 bluefs umount 2026-02-20T16:28:33.088 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.084+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) close 2026-02-20T16:28:33.151 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-20T16:28:33.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-20T16:28:33.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-20T16:28:33.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount 2026-02-20T16:28:33.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-20T16:28:33.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-20T16:28:33.154 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-20T16:28:33.154 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.148+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-20T16:28:33.164 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.160+0000 7fbd26c86d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-20T16:28:33.165 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-20T16:28:33.165 INFO:tasks.workunit.client.0.trial013.stdout:00000000 0b 05 58 02 00 00 45 f1 f7 30 fe 23 45 e2 82 e9 |..X...E..0.#E...| 2026-02-20T16:28:33.165 INFO:tasks.workunit.client.0.trial013.stdout:00000010 a6 20 74 37 61 96 02 00 00 00 1a 00 00 00 00 00 |. t7a...........| 2026-02-20T16:28:33.165 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:* 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-20T16:28:33.166 INFO:tasks.workunit.client.0.trial013.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-20T16:28:33.167 INFO:tasks.workunit.client.0.trial013.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000220 72 65 1a 00 00 00 0d 00 00 00 69 70 67 cf a7 8a |re........ipg...| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000230 4a 0e a7 0c a8 be 39 ea 62 af 00 00 00 00 00 00 |J.....9.b.......| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000240 00 00 1a 00 00 00 9d 8b 98 69 ba f3 d4 2b 01 00 |.........i...+..| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1a 00 00 00 |..............| 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout:0000025e 2026-02-20T16:28:33.168 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:33.169 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:0000003f 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-20T16:28:33.170 INFO:tasks.workunit.client.0.trial013.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000047 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:33.171 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:33.172 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:0000003b 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.173 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-20T16:28:33.174 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.174 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-20T16:28:33.174 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:33.174 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.160+0000 7fbd26c86d40 1 bluefs umount 2026-02-20T16:28:33.174 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.160+0000 7fbd26c86d40 1 bdev(0x55b570901c00 td/osd-mapper/2/block) close 2026-02-20T16:28:33.219 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.216+0000 7fbd26c86d40 1 freelist shutdown 2026-02-20T16:28:33.219 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:33.216+0000 7fbd26c86d40 1 bdev(0x55b570901800 td/osd-mapper/2/block) close 2026-02-20T16:28:33.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-20T16:28:33.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-20T16:28:33.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:33.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:33.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:33.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-20T16:28:33.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-20T16:28:33.517 INFO:tasks.workunit.client.0.trial013.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:33.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-20T16:28:33.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.3IlXhCEw7O_the_val 2026-02-20T16:28:33.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:33.518 INFO:tasks.workunit.client.0.trial013.stdout:Value dumped in: /tmp/tmp.3IlXhCEw7O_the_val 2026-02-20T16:28:33.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.3IlXhCEw7O_the_val' 2026-02-20T16:28:33.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.3IlXhCEw7O_the_val 2026-02-20T16:28:33.660 INFO:tasks.workunit.client.0.trial013.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-20T16:28:33.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:33.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.3IlXhCEw7O_the_val 2026-02-20T16:28:33.731 INFO:tasks.workunit.client.0.trial013.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-20T16:28:33.731 INFO:tasks.workunit.client.0.trial013.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-20T16:28:33.731 INFO:tasks.workunit.client.0.trial013.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-20T16:28:33.732 INFO:tasks.workunit.client.0.trial013.stdout:0000073 2026-02-20T16:28:33.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-20T16:28:33.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:33.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.3IlXhCEw7O_the_val 2026-02-20T16:28:34.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:34.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 ' 2026-02-20T16:28:34.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 2026-02-20T16:28:34.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:34.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:34.188 INFO:tasks.workunit.client.0.trial013.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ 2026-02-20T16:28:34.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/' 2026-02-20T16:28:34.189 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2026-02-20T16:28:34.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:34.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:34.193 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:28:34.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:28:34.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2026-02-20T16:28:34.195 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:28:34.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:28:34.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:28:34.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2026-02-20T16:28:34.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:28:34.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:28:34.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:28:34.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:28:34.226 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:34.220+0000 7f4f3033aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:34.227 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:34.224+0000 7f4f3033aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:34.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:34.224+0000 7f4f3033aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:34.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:28:34.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:28:34.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:28:34.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:28:34.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:28:34.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:34.363 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:34.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:28:34.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:34.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:28:34.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:34.617 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:34.612+0000 7f4f3033aa40 -1 Falling back to public interface 2026-02-20T16:28:35.250 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:35.244+0000 7f4f3033aa40 -1 osd.0 25 log_to_monitors true 2026-02-20T16:28:35.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:28:35.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:35.532 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:35.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:28:35.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:35.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:28:35.699 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6800/3203198978,v1:127.0.0.1:6801/3203198978] [v2:127.0.0.1:6802/3203198978,v1:127.0.0.1:6803/3203198978] exists,up c31287ff-3fd4-4416-a367-ef0b71d0cbf8 2026-02-20T16:28:35.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:28:35.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2026-02-20T16:28:35.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:35.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:28:35.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:28:35.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2026-02-20T16:28:35.706 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:28:35.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:28:35.728 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:35.724+0000 7fa064e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:35.730 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:35.724+0000 7fa064e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:35.730 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:35.724+0000 7fa064e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:35.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:28:35.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:28:35.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:28:35.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:28:35.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:28:35.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:35.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:28:35.879 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:28:36.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:36.172 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:36.168+0000 7fa064e64a40 -1 Falling back to public interface 2026-02-20T16:28:36.866 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:36.860+0000 7fa064e64a40 -1 osd.1 26 log_to_monitors true 2026-02-20T16:28:37.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:28:37.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:37.051 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:37.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:28:37.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:37.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 26 last_clean_interval [9,26) [v2:127.0.0.1:6810/1568791897,v1:127.0.0.1:6811/1568791897] [v2:127.0.0.1:6812/1568791897,v1:127.0.0.1:6813/1568791897] exists,up 1af5d89d-168e-4f6d-a358-0967676f6a48 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-20T16:28:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:37.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:28:37.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:28:37.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=45f1f730-fe23-45e2-82e9-a62074376196 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12019/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:28:37.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2026-02-20T16:28:37.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:28:37.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:28:37.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:28:37.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:28:37.260 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:37.256+0000 7f3f31db4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:37.262 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:37.256+0000 7f3f31db4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:37.263 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:37.260+0000 7f3f31db4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:28:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:28:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:28:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:28:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:28:37.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:37.398 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:37.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:28:37.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:37.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:28:37.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:37.720 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:37.716+0000 7f3f31db4a40 -1 Falling back to public interface 2026-02-20T16:28:38.258 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:38.252+0000 7f3f31db4a40 -1 osd.2 26 log_to_monitors true 2026-02-20T16:28:38.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:28:38.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:38.564 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:38.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:28:38.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:38.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:28:38.734 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 35 up_thru 35 down_at 26 last_clean_interval [13,26) [v2:127.0.0.1:6818/1085512523,v1:127.0.0.1:6819/1085512523] [v2:127.0.0.1:6820/1085512523,v1:127.0.0.1:6821/1085512523] exists,up 697067cf-a78a-4a0e-a70c-a8be39ea62af 2026-02-20T16:28:38.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:28:38.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:28:38.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:28:38.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2026-02-20T16:28:39.737 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2026-02-20T16:28:39.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2026-02-20T16:28:39.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:39.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2026-02-20T16:28:39.803 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:39.804 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T16:28:39.804 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T16:28:39.804 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T16:28:39.804 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:39.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:39.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2026-02-20T16:28:39.874 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:39.875 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T16:28:39.875 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T16:28:39.875 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T16:28:39.875 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:39.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-20T16:28:39.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2026-02-20T16:28:39.943 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:39.944 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T16:28:39.944 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T16:28:39.944 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T16:28:39.944 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:39.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: "name": "objxxx", 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 5, 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: "id": 2, 2026-02-20T16:28:39.969 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "id": 1, 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap01" 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "id": 2, 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap02" 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6, 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:39.970 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "id": 3, 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "id": 3, 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap13" 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6, 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.971 INFO:tasks.workunit.client.0.trial013.stdout: "id": 5, 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [ 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "id": 4, 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap24" 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "id": 5, 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "name": "snap25" 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "size": 5, 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: "overlaps": [] 2026-02-20T16:28:39.972 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: "id": "head", 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: "snapshots": [], 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: "size": 6 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T16:28:39.973 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:39.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2026-02-20T16:28:40.184 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is unset 2026-02-20T16:28:40.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2026-02-20T16:28:40.388 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is unset 2026-02-20T16:28:40.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-20T16:28:40.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-20T16:28:40.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2026-02-20T16:28:40.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-20T16:28:40.706 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:28:40.706 INFO:tasks.workunit.client.0.trial013.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-20T16:28:38.512637+0000 23'7 36:70 [1,2,0] 1 [1,2,0] 1 23'7 2026-02-20T16:28:20.800226+0000 23'7 2026-02-20T16:28:20.800226+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:28:20.800226+0000 4 0 2026-02-20T16:28:40.706 INFO:tasks.workunit.client.0.trial013.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512531+0000 0'0 36:41 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:40.706 INFO:tasks.workunit.client.0.trial013.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512482+0000 0'0 36:51 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:40.706 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512501+0000 0'0 36:60 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:40.707 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:40.707 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:28:40.707 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:28:40.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2026-02-20T16:28:42.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-20T16:28:42.779 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:42.780 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:28:42.780 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T16:28:42.780 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T16:28:42.780 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:42.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2026-02-20T16:28:47.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-20T16:28:42.848644+0000 23'7 38:84 [1,2,0] 1 [1,2,0] 1 23'7 2026-02-20T16:28:42.848606+0000 23'7 2026-02-20T16:28:42.848606+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:28:42.848606+0000 4 0 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512531+0000 0'0 38:46 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512482+0000 0'0 38:55 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512501+0000 0'0 38:66 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:47.931 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:28:47.932 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:28:47.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:47.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:47.943 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:28:42.844+0000 7fa040dde640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-20T16:28:47.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:47.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:47.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2026-02-20T16:28:47.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2026-02-20T16:28:47.948 INFO:tasks.workunit.client.0.trial013.stdout:prev count: 1 2026-02-20T16:28:47.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2026-02-20T16:28:47.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-20T16:28:48.019 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:28:48.019 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:28:48.019 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T16:28:48.019 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T16:28:48.019 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:28:48.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2026-02-20T16:28:53.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-20T16:28:53.173 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:28:53.173 INFO:tasks.workunit.client.0.trial013.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-20T16:28:48.756682+0000 23'7 38:93 [1,2,0] 1 [1,2,0] 1 23'7 2026-02-20T16:28:48.756632+0000 23'7 2026-02-20T16:28:48.756632+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:28:48.756632+0000 4 0 2026-02-20T16:28:53.173 INFO:tasks.workunit.client.0.trial013.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512531+0000 0'0 38:47 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:53.173 INFO:tasks.workunit.client.0.trial013.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512482+0000 0'0 38:56 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:53.173 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:28:38.512501+0000 0'0 38:67 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-20T16:28:14.351366+0000 0'0 2026-02-20T16:28:14.351366+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:28:14.351366+0000 0 0 2026-02-20T16:28:53.174 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:28:53.174 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:28:53.174 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:28:53.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:53.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:53.185 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:28:42.844+0000 7fa040dde640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-20T16:28:53.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-20T16:28:53.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2026-02-20T16:28:53.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2026-02-20T16:28:53.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-20T16:28:53.189 INFO:tasks.workunit.client.0.trial013.stdout:current count: 1 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:53.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:53.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-20T16:28:53.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-20T16:28:53.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-20T16:28:53.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2026-02-20T16:28:53.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:53.706 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:53.707 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-20T16:28:53.707 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-20T16:28:53.707 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-20T16:28:53.707 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:53.707 INFO:tasks.workunit.client.0.trial013.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-20T16:28:53.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-20T16:28:53.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-20T16:28:53.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-20T16:28:53.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2026-02-20T16:28:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2026-02-20T16:28:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2026-02-20T16:28:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:53.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:54.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:54.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:28:54.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:28:54.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:28:54.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:28:54.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:28:54.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:28:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:28:54.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:28:54.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:28:54.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:28:54.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-20T16:28:54.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:28:54.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:54.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12019 2026-02-20T16:28:54.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-mapper 0 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:54.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:54.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:28:54.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:28:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:28:54.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:28:54.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:28:54.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:28:54.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:28:54.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:28:54.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:28:54.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T16:28:54.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-20T16:28:54.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:28:54.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12019 2026-02-20T16:28:54.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12019 2026-02-20T16:28:54.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:28:54.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:28:54.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T16:28:54.093 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T16:28:54.093 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T16:28:54.143 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2026-02-20T16:28:54.143 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2026-02-20T16:28:54.196 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T16:28:54.199 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T16:28:54.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:28:54.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-recovery-scrub 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2026-02-20T16:28:54.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2026-02-20T16:28:54.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2026-02-20T16:28:54.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none ' 2026-02-20T16:28:54.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-20T16:28:54.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2026-02-20T16:28:54.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T16:28:54.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2026-02-20T16:28:54.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local funcs=TEST_recovery_scrub_1 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.30505 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2026-02-20T16:28:54.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:28:54.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:28:54.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:28:54.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:28:54.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:28:54.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:28:54.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:28:54.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:28:54.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:28:54.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:28:54.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:28:54.209 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:28:54.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:28:54.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:28:54.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-20T16:28:54.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:28:54.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30505 2026-02-20T16:28:54.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:28:54.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:28:54.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2026-02-20T16:28:54.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:28:54.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.30505 2026-02-20T16:28:54.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:28:54.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:28:54.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:28:54.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:28:54.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2026-02-20T16:28:54.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:54.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:28:54.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:28:54.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:28:54.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:28:54.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:28:54.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:28:54.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:28:54.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:28:54.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30505/ceph-mon.a.asok 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:28:54.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30505/ceph-mon.a.asok config get fsid 2026-02-20T16:28:54.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:28:54.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:28:54.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:28:54.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30505/ceph-mon.a.asok 2026-02-20T16:28:54.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:28:54.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30505/ceph-mon.a.asok config get mon_host 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2026-02-20T16:28:54.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:28:54.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:28:54.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:54.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:54.483 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:54.483 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:54.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:28:54.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:28:54.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2026-02-20T16:28:54.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2026-02-20T16:28:54.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2026-02-20T16:28:54.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-20T16:28:54.504 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2026-02-20T16:28:54.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2026-02-20T16:28:54.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:28:54.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:54.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:28:54.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2026-02-20T16:28:54.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:28:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c 2026-02-20T16:28:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c' 2026-02-20T16:28:54.512 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c 2026-02-20T16:28:54.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:28:54.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDGi5hpzFzeHhAAdxqSzSy7ZER8U/+yQSGewA== 2026-02-20T16:28:54.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDGi5hpzFzeHhAAdxqSzSy7ZER8U/+yQSGewA=="}' 2026-02-20T16:28:54.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c -i td/osd-recovery-scrub/0/new.json 2026-02-20T16:28:54.605 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2026-02-20T16:28:54.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDGi5hpzFzeHhAAdxqSzSy7ZER8U/+yQSGewA== --osd-uuid b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c 2026-02-20T16:28:54.629 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:54.623+0000 7fc685691a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:54.631 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:54.627+0000 7fc685691a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:54.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:54.627+0000 7fc685691a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:54.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:54.627+0000 7fc685691a40 -1 bdev(0x564ab0420c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:28:54.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:54.627+0000 7fc685691a40 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2026-02-20T16:28:55.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2026-02-20T16:28:55.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:28:55.319 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:28:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:28:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:28:55.414 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:28:55.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:28:55.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-20T16:28:55.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:28:55.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:28:55.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:28:55.429 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:55.423+0000 7f7b7cad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:55.430 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:55.423+0000 7f7b7cad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:55.431 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:55.427+0000 7f7b7cad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:55.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:28:55.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:28:55.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:28:55.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:28:55.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:28:55.509 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:55.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:55.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:28:55.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:55.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:28:55.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:55.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:55.859+0000 7f7b7cad1a40 -1 Falling back to public interface 2026-02-20T16:28:56.517 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:56.511+0000 7f7b7cad1a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:28:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:28:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:56.601 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:28:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:28:56.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:57.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:28:57.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:57.719 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:28:57.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:28:57.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:57.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:28:57.880 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1413950927,v1:127.0.0.1:6801/1413950927] [v2:127.0.0.1:6802/1413950927,v1:127.0.0.1:6803/1413950927] exists,up b7b4e0bd-865d-4513-a3a1-faaa0c5ff40c 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2026-02-20T16:28:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2026-02-20T16:28:57.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:28:57.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2026-02-20T16:28:57.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:28:57.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ce95e690-72f8-4ce5-af9d-b6f75db14d0d 2026-02-20T16:28:57.887 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 ce95e690-72f8-4ce5-af9d-b6f75db14d0d 2026-02-20T16:28:57.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ce95e690-72f8-4ce5-af9d-b6f75db14d0d' 2026-02-20T16:28:57.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:28:57.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJi5hp1eNmNRAAnCUTFAoWwsoIh2d5xe2HmA== 2026-02-20T16:28:57.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJi5hp1eNmNRAAnCUTFAoWwsoIh2d5xe2HmA=="}' 2026-02-20T16:28:57.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ce95e690-72f8-4ce5-af9d-b6f75db14d0d -i td/osd-recovery-scrub/1/new.json 2026-02-20T16:28:58.051 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:28:58.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2026-02-20T16:28:58.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDJi5hp1eNmNRAAnCUTFAoWwsoIh2d5xe2HmA== --osd-uuid ce95e690-72f8-4ce5-af9d-b6f75db14d0d 2026-02-20T16:28:58.078 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.071+0000 7fd86973fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:58.079 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.075+0000 7fd86973fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:58.080 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.075+0000 7fd86973fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:58.081 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.075+0000 7fd86973fa40 -1 bdev(0x559992da4c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:28:58.081 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.075+0000 7fd86973fa40 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2026-02-20T16:28:58.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2026-02-20T16:28:58.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:28:58.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:28:58.717 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:28:58.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:28:58.927 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:28:58.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:28:58.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-20T16:28:58.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:28:58.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:28:58.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:28:58.950 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.943+0000 7f09de983a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:58.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.947+0000 7f09de983a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:58.953 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:58.947+0000 7f09de983a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:28:59.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:28:59.085 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:28:59.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:28:59.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:28:59.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:28:59.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:28:59.300 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:59.295+0000 7f09de983a40 -1 Falling back to public interface 2026-02-20T16:28:59.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:28:59.947+0000 7f09de983a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:29:00.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:29:00.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:29:00.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:29:00.247 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:00.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:29:00.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:29:00.403 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2886987655,v1:127.0.0.1:6811/2886987655] [v2:127.0.0.1:6812/2886987655,v1:127.0.0.1:6813/2886987655] exists,up ce95e690-72f8-4ce5-af9d-b6f75db14d0d 2026-02-20T16:29:00.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:29:00.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:29:00.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:29:00.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:29:00.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2026-02-20T16:29:00.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:29:00.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=378b8b2c-d254-4c21-8591-c12e320cb513 2026-02-20T16:29:00.408 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 378b8b2c-d254-4c21-8591-c12e320cb513 2026-02-20T16:29:00.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 378b8b2c-d254-4c21-8591-c12e320cb513' 2026-02-20T16:29:00.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:29:00.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMi5hp3DYDGRAA+HDt5gMvv7MMvymVTpbBpA== 2026-02-20T16:29:00.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMi5hp3DYDGRAA+HDt5gMvv7MMvymVTpbBpA=="}' 2026-02-20T16:29:00.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 378b8b2c-d254-4c21-8591-c12e320cb513 -i td/osd-recovery-scrub/2/new.json 2026-02-20T16:29:00.572 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:29:00.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2026-02-20T16:29:00.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDMi5hp3DYDGRAA+HDt5gMvv7MMvymVTpbBpA== --osd-uuid 378b8b2c-d254-4c21-8591-c12e320cb513 2026-02-20T16:29:00.608 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:00.603+0000 7f23f484da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:00.609 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:00.603+0000 7f23f484da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:00.610 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:00.603+0000 7f23f484da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:00.611 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:00.607+0000 7f23f484da40 -1 bdev(0x56174b5b6c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:29:00.611 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:00.607+0000 7f23f484da40 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2026-02-20T16:29:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2026-02-20T16:29:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:29:01.274 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:29:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:29:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:29:01.472 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:29:01.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:29:01.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-20T16:29:01.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:29:01.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:29:01.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:29:01.496 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:01.491+0000 7f7415a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:01.497 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:01.491+0000 7f7415a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:01.498 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:01.491+0000 7f7415a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:01.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:29:01.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:29:01.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:29:01.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:29:01.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:29:01.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:29:01.628 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:29:01.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:29:01.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:29:01.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:29:01.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:29:01.896 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:01.891+0000 7f7415a8fa40 -1 Falling back to public interface 2026-02-20T16:29:02.528 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:02.523+0000 7f7415a8fa40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:29:02.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:29:02.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:29:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:29:02.791 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:29:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:29:02.949 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/902639939,v1:127.0.0.1:6819/902639939] [v2:127.0.0.1:6820/902639939,v1:127.0.0.1:6821/902639939] exists,up 378b8b2c-d254-4c21-8591-c12e320cb513 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:29:02.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:29:02.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:29:02.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:29:02.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:29:02.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-20T16:29:02.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2026-02-20T16:29:02.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:29:02.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d736d004-1a0d-48b4-a767-3d87327a51b6 2026-02-20T16:29:02.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 d736d004-1a0d-48b4-a767-3d87327a51b6' 2026-02-20T16:29:02.954 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 d736d004-1a0d-48b4-a767-3d87327a51b6 2026-02-20T16:29:02.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:29:02.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOi5hph5aCORAAmC8QGmQw8xn/J8DayPugPw== 2026-02-20T16:29:02.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOi5hph5aCORAAmC8QGmQw8xn/J8DayPugPw=="}' 2026-02-20T16:29:02.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d736d004-1a0d-48b4-a767-3d87327a51b6 -i td/osd-recovery-scrub/3/new.json 2026-02-20T16:29:03.126 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T16:29:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2026-02-20T16:29:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDOi5hph5aCORAAmC8QGmQw8xn/J8DayPugPw== --osd-uuid d736d004-1a0d-48b4-a767-3d87327a51b6 2026-02-20T16:29:03.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:03.147+0000 7fd6e3b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:03.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:03.147+0000 7fd6e3b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:03.155 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:03.147+0000 7fd6e3b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:03.155 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:03.151+0000 7fd6e3b0ba40 -1 bdev(0x558334242c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2026-02-20T16:29:03.155 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:03.151+0000 7fd6e3b0ba40 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2026-02-20T16:29:03.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2026-02-20T16:29:03.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:29:03.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T16:29:03.823 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T16:29:03.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:29:04.030 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T16:29:04.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T16:29:04.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=f1bc0b4a-72c4-4203-8788-bd3be4d025a5 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.30505/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-20T16:29:04.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:29:04.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:29:04.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:29:04.054 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:04.047+0000 7fd59cc81a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:04.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:04.051+0000 7fd59cc81a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:04.056 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:04.051+0000 7fd59cc81a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:04.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T16:29:04.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:29:04.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:29:04.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:29:04.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:29:04.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:29:04.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:29:04.187 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:29:04.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:29:04.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:29:04.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:29:04.436 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:04.431+0000 7fd59cc81a40 -1 Falling back to public interface 2026-02-20T16:29:04.992 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:04.987+0000 7fd59cc81a40 -1 osd.3 0 log_to_monitors true 2026-02-20T16:29:05.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:29:05.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:29:05.347 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:05.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:29:05.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:29:05.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:29:05.505 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1780745403,v1:127.0.0.1:6827/1780745403] [v2:127.0.0.1:6828/1780745403,v1:127.0.0.1:6829/1780745403] exists,up d736d004-1a0d-48b4-a767-3d87327a51b6 2026-02-20T16:29:05.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:29:05.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:29:05.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:29:05.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2026-02-20T16:29:05.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T16:29:05.711 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T16:29:05.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:29:06.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2026-02-20T16:29:06.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:29:06.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:29:06.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:29:06.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:29:06.724 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:29:06.724 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:29:06.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:29:06.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:29:06.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:29:06.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:29:06.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:29:06.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:29:06.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:29:06.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:29:06.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:29:06.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:29:06.987 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:29:06.987 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:29:06.988 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:29:06.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:29:06.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:06.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:29:07.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T16:29:07.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T16:29:07.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T16:29:07.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:07.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:29:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:29:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:29:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-20T16:29:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:07.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:29:07.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-20T16:29:07.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-20T16:29:07.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574850' 2026-02-20T16:29:07.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:07.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:29:07.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444034 2026-02-20T16:29:07.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444034 2026-02-20T16:29:07.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574850 3-73014444034' 2026-02-20T16:29:07.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:07.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T16:29:07.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:07.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:29:07.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T16:29:07.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:07.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T16:29:07.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T16:29:07.284 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T16:29:07.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:29:07.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-20T16:29:07.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:29:08.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:29:08.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:29:08.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-20T16:29:08.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:29:09.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:29:09.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:29:09.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T16:29:09.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:29:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:09.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:29:09.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:29:09.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:09.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:29:09.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:29:09.762 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:29:09.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:29:09.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:29:09.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:09.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-20T16:29:09.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:09.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:29:09.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-20T16:29:09.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:09.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-20T16:29:09.935 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574850 2026-02-20T16:29:09.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-20T16:29:09.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:29:10.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574850 2026-02-20T16:29:10.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:10.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444034 2026-02-20T16:29:10.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:10.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:29:10.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444034 2026-02-20T16:29:10.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:10.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444034 2026-02-20T16:29:10.093 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 73014444034 2026-02-20T16:29:10.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444034' 2026-02-20T16:29:10.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:29:10.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444034 -lt 73014444034 2026-02-20T16:29:10.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:29:10.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:29:10.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:29:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:29:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:29:10.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:29:10.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:29:10.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:29:10.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:29:10.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:29:10.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:29:10.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:29:10.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:29:10.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:29:10.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:29:10.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:29:10.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:29:10.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:29:10.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2026-02-20T16:29:10.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2026-02-20T16:29:10.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2026-02-20T16:29:10.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2026-02-20T16:29:10.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-20T16:29:11.103 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:29:11.104 INFO:tasks.workunit.client.0.trial013.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-20T16:29:05.916412+0000 0'0 20:12 [1] 1 [1] 1 0'0 2026-02-20T16:29:05.710313+0000 0'0 2026-02-20T16:29:05.710313+0000 0 0 periodic scrub scheduled @ 2026-02-22T03:24:40.375051+0000 0 0 2026-02-20T16:29:11.104 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:29:11.104 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:29:11.104 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:11.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.30505 bs=1M count=50 2026-02-20T16:29:11.240 INFO:tasks.workunit.client.0.trial013.stderr:50+0 records in 2026-02-20T16:29:11.241 INFO:tasks.workunit.client.0.trial013.stderr:50+0 records out 2026-02-20T16:29:11.241 INFO:tasks.workunit.client.0.trial013.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.125891 s, 416 MB/s 2026-02-20T16:29:11.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2026-02-20T16:29:11.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.30505 2026-02-20T16:29:11.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.30505 2026-02-20T16:29:11.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.30505 2026-02-20T16:29:11.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.30505 2026-02-20T16:29:11.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.30505 2026-02-20T16:29:11.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.30505 2026-02-20T16:29:11.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.30505 2026-02-20T16:29:11.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.30505 2026-02-20T16:29:11.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.30505 2026-02-20T16:29:11.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:11.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.30505 2026-02-20T16:29:12.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.30505 2026-02-20T16:29:12.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.30505 2026-02-20T16:29:12.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.30505 2026-02-20T16:29:12.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.30505 2026-02-20T16:29:12.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.30505 2026-02-20T16:29:12.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.30505 2026-02-20T16:29:12.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.30505 2026-02-20T16:29:12.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.30505 2026-02-20T16:29:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.30505 2026-02-20T16:29:12.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.30505 2026-02-20T16:29:12.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.30505 2026-02-20T16:29:12.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:12.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.30505 2026-02-20T16:29:13.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.30505 2026-02-20T16:29:13.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.30505 2026-02-20T16:29:13.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.30505 2026-02-20T16:29:13.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.30505 2026-02-20T16:29:13.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.30505 2026-02-20T16:29:13.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.30505 2026-02-20T16:29:13.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.30505 2026-02-20T16:29:13.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.30505 2026-02-20T16:29:13.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.30505 2026-02-20T16:29:13.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.30505 2026-02-20T16:29:13.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.30505 2026-02-20T16:29:13.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.30505 2026-02-20T16:29:13.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:13.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.30505 2026-02-20T16:29:14.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.30505 2026-02-20T16:29:14.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.30505 2026-02-20T16:29:14.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.30505 2026-02-20T16:29:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.30505 2026-02-20T16:29:14.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.30505 2026-02-20T16:29:14.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.30505 2026-02-20T16:29:14.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.30505 2026-02-20T16:29:14.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.30505 2026-02-20T16:29:14.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.30505 2026-02-20T16:29:14.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.30505 2026-02-20T16:29:14.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.30505 2026-02-20T16:29:14.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:14.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.30505 2026-02-20T16:29:15.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.30505 2026-02-20T16:29:15.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.30505 2026-02-20T16:29:15.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.30505 2026-02-20T16:29:15.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.30505 2026-02-20T16:29:15.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.30505 2026-02-20T16:29:15.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.30505 2026-02-20T16:29:15.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.30505 2026-02-20T16:29:15.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.30505 2026-02-20T16:29:15.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.30505 2026-02-20T16:29:15.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.30505 2026-02-20T16:29:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.30505 2026-02-20T16:29:15.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:15.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.30505 2026-02-20T16:29:16.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.30505 2026-02-20T16:29:16.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.30505 2026-02-20T16:29:16.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.30505 2026-02-20T16:29:16.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.30505 2026-02-20T16:29:16.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.30505 2026-02-20T16:29:16.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.30505 2026-02-20T16:29:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.30505 2026-02-20T16:29:16.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.30505 2026-02-20T16:29:16.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.30505 2026-02-20T16:29:16.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.30505 2026-02-20T16:29:16.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.30505 2026-02-20T16:29:16.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:16.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.30505 2026-02-20T16:29:17.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.30505 2026-02-20T16:29:17.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.30505 2026-02-20T16:29:17.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.30505 2026-02-20T16:29:17.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.30505 2026-02-20T16:29:17.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.30505 2026-02-20T16:29:17.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.30505 2026-02-20T16:29:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.30505 2026-02-20T16:29:17.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.30505 2026-02-20T16:29:17.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.30505 2026-02-20T16:29:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.30505 2026-02-20T16:29:17.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.30505 2026-02-20T16:29:17.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:17.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.30505 2026-02-20T16:29:18.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.30505 2026-02-20T16:29:18.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.30505 2026-02-20T16:29:18.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.30505 2026-02-20T16:29:18.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.30505 2026-02-20T16:29:18.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.30505 2026-02-20T16:29:18.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.30505 2026-02-20T16:29:18.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.30505 2026-02-20T16:29:18.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.30505 2026-02-20T16:29:18.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.30505 2026-02-20T16:29:18.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.30505 2026-02-20T16:29:18.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:18.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.30505 2026-02-20T16:29:19.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.30505 2026-02-20T16:29:19.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.30505 2026-02-20T16:29:19.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.30505 2026-02-20T16:29:19.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.30505 2026-02-20T16:29:19.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.30505 2026-02-20T16:29:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-20T16:29:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.30505 2026-02-20T16:29:19.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.30505 2026-02-20T16:29:19.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2026-02-20T16:29:19.722 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 4 2026-02-20T16:29:19.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2026-02-20T16:29:19.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2026-02-20T16:29:19.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-20T16:29:19.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-20T16:29:19.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-20T16:29:19.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-20T16:29:19.881 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:19.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-20T16:29:21.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2026-02-20T16:29:21.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2026-02-20T16:29:21.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2026-02-20T16:29:21.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-20T16:29:21.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-20T16:29:21.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-20T16:29:21.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-20T16:29:22.048 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:22.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-20T16:29:24.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2026-02-20T16:29:24.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2026-02-20T16:29:24.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2026-02-20T16:29:24.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-20T16:29:24.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-20T16:29:24.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-20T16:29:24.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-20T16:29:24.222 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:24.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-20T16:29:26.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2026-02-20T16:29:26.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2026-02-20T16:29:26.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2026-02-20T16:29:26.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-20T16:29:26.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-20T16:29:26.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-20T16:29:26.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-20T16:29:26.399 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:26.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-20T16:29:28.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2026-02-20T16:29:28.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2026-02-20T16:29:28.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2026-02-20T16:29:28.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-20T16:29:28.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-20T16:29:28.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-20T16:29:28.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-20T16:29:28.577 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:28.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2026-02-20T16:29:28.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2026-02-20T16:29:28.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-20T16:29:28.736 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:29:28.736 INFO:tasks.workunit.client.0.trial013.stdout:1.0 100 0 243 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2026-02-20T16:29:22.011853+0000 23'1300 26:1404 [1,0,2,3] 1 [1,0] 1 0'0 2026-02-20T16:29:05.710313+0000 0'0 2026-02-20T16:29:05.710313+0000 0 0 no scrub is scheduled 0 0 2026-02-20T16:29:28.736 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:29:28.737 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:29:28.737 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:29:28.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2026-02-20T16:29:38.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2026-02-20T16:29:38.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:29:38.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:29:38.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:29:38.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:29:38.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:29:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:29:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2026-02-20T16:29:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2026-02-20T16:29:44.074 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2026-02-20T16:29:44.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2026-02-20T16:29:44.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-20T16:29:44.081 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:22.435+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.081 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:23.407+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:24.383+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:25.399+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:26.403+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:27.411+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:28.379+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:29.331+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:30.355+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:31.395+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:32.403+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:33.387+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.082 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:34.427+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:35.447+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:36.407+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:37.431+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:38.471+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:39.427+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:40.475+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:41.523+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.083 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:42.535+0000 7f7b75284640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-20T16:29:44.097 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:22.971+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.097 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:24.003+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.097 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:24.951+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.097 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:25.931+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.097 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:26.951+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:27.959+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:28.963+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:29.999+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:30.991+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:32.003+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:33.051+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:34.091+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:35.091+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:36.131+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:37.087+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:38.055+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:39.011+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.098 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:40.015+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.099 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:41.063+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.099 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:42.103+0000 7f09d7136640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-20T16:29:44.101 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:22.643+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.101 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:23.635+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.101 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:24.643+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:25.607+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:26.559+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:27.527+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:28.551+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:29.539+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:30.579+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:31.575+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:32.563+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:33.579+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:34.547+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:35.583+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:36.595+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.102 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:37.567+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:38.611+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:39.571+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:40.547+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:41.595+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:42.639+0000 7f740e242640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:22.975+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:23.975+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:24.939+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:25.907+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:26.923+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:27.915+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:28.879+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:29.927+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:30.883+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.107 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:31.867+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:32.911+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:33.883+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:34.891+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:35.911+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:36.939+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:37.919+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:38.911+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:39.939+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:40.915+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.108 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:29:41.907+0000 7fd595434640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-20T16:29:44.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2026-02-20T16:29:44.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2026-02-20T16:29:44.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2026-02-20T16:29:44.109 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2026-02-20T16:29:44.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2026-02-20T16:29:44.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-20T16:29:44.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-20T16:29:44.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2026-02-20T16:29:44.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2026-02-20T16:29:44.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-20T16:29:44.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-20T16:29:44.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2026-02-20T16:29:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2026-02-20T16:29:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-20T16:29:44.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-20T16:29:44.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2026-02-20T16:29:44.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2026-02-20T16:29:44.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:29:44.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-20T16:29:44.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-20T16:29:44.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:29:44.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:29:44.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:29:44.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:29:44.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:29:44.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:29:44.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:29:44.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:29:44.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:29:44.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:29:44.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:29:44.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:29:44.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:29:44.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:29:44.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:29:44.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:29:44.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-20T16:29:44.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:29:44.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:44.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:29:44.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30505 2026-02-20T16:29:44.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:29:44.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:29:44.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2026-02-20T16:29:44.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2026-02-20T16:29:44.192 INFO:tasks.workunit.client.0.trial013.stdout:TEST PASSED 2026-02-20T16:29:44.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2026-02-20T16:29:44.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T16:29:44.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-recovery-scrub 0 2026-02-20T16:29:44.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:29:44.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:29:44.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:29:44.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:29:44.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:29:44.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:29:44.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:29:44.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:29:44.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:29:44.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:29:44.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:29:44.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:29:44.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T16:29:44.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-20T16:29:44.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:29:44.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:44.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30505 2026-02-20T16:29:44.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30505 2026-02-20T16:29:44.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:29:44.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:29:44.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T16:29:44.204 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T16:29:44.204 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T16:29:44.255 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2026-02-20T16:29:44.255 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2026-02-20T16:29:44.307 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-dump 2026-02-20T16:29:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stdout:This test is disabled 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-dump 0 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:29:44.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:29:44.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:29:44.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:29:44.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:29:44.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:29:44.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:29:44.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:29:44.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:29:44.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:29:44.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:29:44.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:29:44.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T16:29:44.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2026-02-20T16:29:44.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:29:44.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:44.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39906 2026-02-20T16:29:44.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39906 2026-02-20T16:29:44.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:29:44.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:29:44.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T16:29:44.325 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T16:29:44.325 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T16:29:44.377 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2026-02-20T16:29:44.377 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ TIMEOUT=300 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ MAX_TIMEOUT=15 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ PG_NUM=4 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ TMPDIR=/tmp 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-20T16:29:44.425 INFO:tasks.workunit.client.0.trial013.stderr:+++ uname 2026-02-20T16:29:44.426 INFO:tasks.workunit.client.0.trial013.stderr:++ '[' Linux = FreeBSD ']' 2026-02-20T16:29:44.426 INFO:tasks.workunit.client.0.trial013.stderr:++ SED=sed 2026-02-20T16:29:44.426 INFO:tasks.workunit.client.0.trial013.stderr:++ AWK=awk 2026-02-20T16:29:44.426 INFO:tasks.workunit.client.0.trial013.stderr:+++ stty -a 2026-02-20T16:29:44.426 INFO:tasks.workunit.client.0.trial013.stderr:+++ head -1 2026-02-20T16:29:44.427 INFO:tasks.workunit.client.0.trial013.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-20T16:29:44.427 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T16:29:44.428 INFO:tasks.workunit.client.0.trial013.stderr:++ termwidth= 2026-02-20T16:29:44.428 INFO:tasks.workunit.client.0.trial013.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-20T16:29:44.428 INFO:tasks.workunit.client.0.trial013.stderr:++ DIFFCOLOPTS='-y ' 2026-02-20T16:29:44.428 INFO:tasks.workunit.client.0.trial013.stderr:++ KERNCORE=kernel.core_pattern 2026-02-20T16:29:44.428 INFO:tasks.workunit.client.0.trial013.stderr:++ EXTRA_OPTS= 2026-02-20T16:29:44.430 INFO:tasks.workunit.client.0.trial013.stderr:++ test '' = TESTS 2026-02-20T16:29:44.430 INFO:tasks.workunit.client.0.trial013.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2026-02-20T16:29:44.430 INFO:tasks.workunit.client.0.trial013.stderr:++ uname 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:+ '[' Linux = FreeBSD ']' 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:+ use_ec_overwrite=true 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:+ getjson=no 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:+ jqfilter='def walk(f): 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:29:44.432 INFO:tasks.workunit.client.0.trial013.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:29:44.437 INFO:tasks.workunit.client.0.trial013.stderr:+ main osd-scrub-repair 2026-02-20T16:29:44.437 INFO:tasks.workunit.client.0.trial013.stderr:+ local dir=td/osd-scrub-repair 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:+ shift 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:+ shopt -s -o xtrace 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T16:29:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-repair 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2026-02-20T16:29:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2026-02-20T16:29:44.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2026-02-20T16:29:44.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none ' 2026-02-20T16:29:44.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2026-02-20T16:29:44.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2026-02-20T16:29:44.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T16:29:44.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2026-02-20T16:29:44.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_oper_initiated_scrub_during_recovery 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_allow_repair_during_recovery 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_bluestore_basic 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_bluestore_failed 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_bluestore_failed_norecov 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_bluestore_scrub 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_bluestore_tag 2026-02-20T16:29:44.444 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_erasure_coded_appends 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_auto_repair_erasure_coded_overwrites 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_and_repair_jerasure_appends 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_and_repair_lrc_appends 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_and_repair_lrc_overwrites 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_and_repair_replicated 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_scrub_erasure_appends 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_scrub_erasure_overwrites 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_scrub_replicated 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_corrupt_snapset_scrub_rep 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_dual_store_replicated_cluster 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_list_missing_erasure_coded_appends 2026-02-20T16:29:44.445 INFO:tasks.workunit.client.0.trial013.stderr:TEST_list_missing_erasure_coded_overwrites 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_periodic_scrub_replicated 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_repair_stats 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_repair_stats_ec 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_request_scrub_priority 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_warning 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_skip_non_repair_during_recovery 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_unfound_erasure_coded_appends 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:TEST_unfound_erasure_coded_overwrites' 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:29:44.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:29:44.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:29:44.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:29:44.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:29:44.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:29:44.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:29:44.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:29:44.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:29:44.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:29:44.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:29:44.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:29:44.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:44.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:29:44.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:29:44.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:29:44.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:29:44.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:29:44.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:29:44.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:29:44.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:29:44.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:29:44.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:29:44.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_oper_initiated_scrub_during_recovery td/osd-scrub-repair 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_oper_initiated_scrub_during_recovery: local dir=td/osd-scrub-repair 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: local -A cluster_conf 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_oper_initiated_scrub_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-20T16:29:44.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T16:29:44.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T16:29:44.753 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 9b8e87fe-15f4-48cf-9409-6d458d5ebb5d 2026-02-20T16:29:44.854 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:29:44.882 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:44.879+0000 7fc1d5f3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:44.884 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:44.879+0000 7fc1d5f3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:44.885 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:44.879+0000 7fc1d5f3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:44.885 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:44.879+0000 7fc1d5f3ea40 -1 bdev(0x561c89ea2c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:29:44.885 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:44.879+0000 7fc1d5f3ea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:29:45.627 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:29:45.731 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:29:45.763 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:45.759+0000 7fa7fa432a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:45.765 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:45.759+0000 7fa7fa432a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:45.767 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:45.763+0000 7fa7fa432a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:45.823 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:29:46.272 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:46.267+0000 7fa7fa432a40 -1 Falling back to public interface 2026-02-20T16:29:46.915 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:47.200 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:47.195+0000 7fa7fa432a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:29:48.012 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:29:48.166 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1030690991,v1:127.0.0.1:6801/1030690991] [v2:127.0.0.1:6802/1030690991,v1:127.0.0.1:6803/1030690991] exists,up 9b8e87fe-15f4-48cf-9409-6d458d5ebb5d 2026-02-20T16:29:48.171 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 4744491a-5f17-4bb1-afc3-e689b2647b9c 2026-02-20T16:29:48.341 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:48.373 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:48.367+0000 7f6b01372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:48.374 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:48.367+0000 7f6b01372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:48.375 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:48.371+0000 7f6b01372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:48.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:48.371+0000 7f6b01372a40 -1 bdev(0x5612558ccc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:29:48.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:48.371+0000 7f6b01372a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:29:49.030 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:29:49.233 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:29:49.264 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:49.258+0000 7fade1b70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:49.266 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:49.262+0000 7fade1b70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:49.268 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:49.262+0000 7fade1b70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:29:49.389 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:29:49.672 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:49.666+0000 7fade1b70a40 -1 Falling back to public interface 2026-02-20T16:29:50.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:29:50.502+0000 7fade1b70a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:29:50.549 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:29:51.709 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:29:51.866 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4015492527,v1:127.0.0.1:6811/4015492527] [v2:127.0.0.1:6812/4015492527,v1:127.0.0.1:6813/4015492527] exists,up 4744491a-5f17-4bb1-afc3-e689b2647b9c 2026-02-20T16:29:52.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_oper_initiated_scrub_during_recovery: local poolname=rbd 2026-02-20T16:29:52.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_oper_initiated_scrub_during_recovery: create_rbd_pool 2026-02-20T16:29:52.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:29:53.019 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:29:53.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:29:53.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:29:53.218 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:29:53.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:29:54.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:29:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_oper_initiated_scrub_during_recovery: wait_for_clean 2026-02-20T16:29:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:29:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:29:54.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:29:54.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:29:54.513 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:29:54.514 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:29:54.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:29:54.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:29:54.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:29:54.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:29:54.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:29:54.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:29:54.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:29:54.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:29:54.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:29:54.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:29:54.781 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:29:54.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:29:54.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:54.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:29:54.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:29:54.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:29:54.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:29:54.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:29:54.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:29:54.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:29:54.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:29:54.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-20T16:29:54.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:54.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:29:54.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:54.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:29:54.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:29:54.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:54.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:29:54.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:29:54.934 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T16:29:54.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:29:55.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:29:55.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:29:55.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:29:55.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:29:55.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:29:55.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:29:55.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:29:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:29:55.103 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:29:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:29:55.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:29:55.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:29:55.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:29:55.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:29:55.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:29:55.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:29:55.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:29:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:29:55.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:29:55.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:29:55.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:29:55.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:29:55.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:29:55.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:29:55.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:29:55.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:130: TEST_allow_oper_initiated_scrub_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:29:55.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:29:55.999 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:29:56.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:29:56.204 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:29:56.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:29:56.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:29:56.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:29:56.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: get_not_primary rbd SOMETHING 2026-02-20T16:29:56.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-20T16:29:56.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:29:56.243 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-20T16:29:56.243 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-20T16:29:56.244 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:29:56.244 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:29:56.244 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:29:56.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:29:56.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:29:56.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:29:56.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: oper_scrub_and_schedule td/osd-scrub-repair rbd 0 2026-02-20T16:29:56.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: oper_scrub_and_schedule: local dir=td/osd-scrub-repair 2026-02-20T16:29:56.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: oper_scrub_and_schedule: local poolname=rbd 2026-02-20T16:29:56.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:182: oper_scrub_and_schedule: local osd=0 2026-02-20T16:29:56.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: get_pg rbd SOMETHING 2026-02-20T16:29:56.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-20T16:29:56.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:29:56.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-20T16:29:56.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:29:56.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: local pg=1.3 2026-02-20T16:29:56.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:29:56.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:29:56.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:29:56.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:29:56.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:29:56.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: local last_scrub=2026-02-20T16:29:53.215669+0000 2026-02-20T16:29:56.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:189: oper_scrub_and_schedule: ceph tell 1.3 scrub 2026-02-20T16:29:56.947 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:29:56.948 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:29:56.948 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T16:29:56.948 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T16:29:56.948 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:29:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: oper_scrub_and_schedule: sleep 3 2026-02-20T16:29:59.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i=0 )) 2026-02-20T16:29:59.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i < 3 )) 2026-02-20T16:29:59.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:29:59.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:29:59.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:29:59.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:29:59.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: test 2026-02-20T16:29:57.642049+0000 '>' 2026-02-20T16:29:53.215669+0000 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: oper_scrub_and_schedule: break 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: oper_scrub_and_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:30:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:00.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-20T16:30:00.500 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:30:00.501 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:30:00.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:00.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:30:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:30:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:30:00.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:30:00.617 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:30:00.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:30:00.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:30:00.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:30:00.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:30:00.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:30:00.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:30:00.633 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:00.630+0000 7f52325b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:00.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:00.634+0000 7f52325b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:00.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:00.634+0000 7f52325b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:00.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:30:00.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:00.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:30:00.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:00.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:00.775 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:30:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:00.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:01.048 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:01.046+0000 7f52325b4a40 -1 Falling back to public interface 2026-02-20T16:30:01.693 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:01.690+0000 7f52325b4a40 -1 osd.0 19 log_to_monitors true 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:01.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/2842403528,v1:127.0.0.1:6801/2842403528] [v2:127.0.0.1:6802/2842403528,v1:127.0.0.1:6803/2842403528] exists,up 9b8e87fe-15f4-48cf-9409-6d458d5ebb5d 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:02.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:02.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:02.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:02.092 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:02.093 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:02.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:02.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:02.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:02.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:02.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:02.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:02.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:02.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:02.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:02.351 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:30:02.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:02.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:02.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:02.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:30:02.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:30:02.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:30:02.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:02.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:02.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-20T16:30:02.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-20T16:30:02.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705677' 2026-02-20T16:30:02.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:02.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:30:02.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:02.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:02.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:30:02.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:02.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:30:02.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:30:02.502 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247810 2026-02-20T16:30:02.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:02.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 98784247810 2026-02-20T16:30:02.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:03.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:03.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:03.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247810 2026-02-20T16:30:03.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:03.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-20T16:30:03.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:03.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:03.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-20T16:30:03.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:03.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-20T16:30:03.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-20T16:30:03.831 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705677 2026-02-20T16:30:03.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:03.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705678 -lt 38654705677 2026-02-20T16:30:03.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:03.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:03.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:04.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:04.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:30:04.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:04.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:04.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:04.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:30:04.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:04.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:04.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: oper_scrub_and_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:30:04.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: oper_scrub_and_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:30:04.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:30:04.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:30:04.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:30:04.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:04.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:04.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:04.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:04.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:04.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:04.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:30:04.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:30:04.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:30:04.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:30:04.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:30:04.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:30:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:30:04.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:30:04.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:04.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:30:04.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:30:04.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:30:04.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:30:04.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:30:04.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:04.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:04.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:04.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:30:04.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:30:04.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:30:04.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:30:04.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:30:04.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:30:04.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:04.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:30:04.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:30:04.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:04.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:30:04.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:30:04.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:30:04.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:30:04.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:30:04.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:30:04.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:30:04.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:30:04.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:30:04.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:30:04.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:30:04.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:30:04.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:138: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_allow_repair_during_recovery: local poolname=rbd 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:30:04.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-20T16:30:04.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:30:04.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:04.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:04.730 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:04.730 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.731 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:04.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T16:30:04.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:30:04.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:30:04.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:30:04.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:30:04.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:30:04.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:30:04.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:30:04.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:30:04.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:30:04.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:30:04.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:30:04.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:30:04.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:30:04.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:30:04.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:30:04.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:142: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:30:04.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:04.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:30:04.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:143: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:04.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:04.989 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:04.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:30:04.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-20T16:30:04.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:30:04.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:30:04.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c593b15-4536-4223-96fd-b2bd8400d910 2026-02-20T16:30:04.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c593b15-4536-4223-96fd-b2bd8400d910' 2026-02-20T16:30:04.992 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 1c593b15-4536-4223-96fd-b2bd8400d910 2026-02-20T16:30:04.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:30:05.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANjJhp+OwKABAAENf81E6inDnXVYCYpC7Z2g== 2026-02-20T16:30:05.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANjJhp+OwKABAAENf81E6inDnXVYCYpC7Z2g=="}' 2026-02-20T16:30:05.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c593b15-4536-4223-96fd-b2bd8400d910 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:30:05.092 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:05.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:30:05.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQANjJhp+OwKABAAENf81E6inDnXVYCYpC7Z2g== --osd-uuid 1c593b15-4536-4223-96fd-b2bd8400d910 2026-02-20T16:30:05.118 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:05.118+0000 7f79258c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:05.120 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:05.118+0000 7f79258c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:05.120 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:05.118+0000 7f79258c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:05.121 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:05.118+0000 7f79258c5a40 -1 bdev(0x561062268c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:30:05.121 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:05.118+0000 7f79258c5a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:30:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:30:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:30:05.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:30:05.911 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:30:05.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:30:06.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:30:06.013 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:30:06.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-20T16:30:06.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:30:06.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:30:06.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:30:06.040 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:06.038+0000 7f93a1f7da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:06.042 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:06.042+0000 7f93a1f7da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:06.044 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:06.042+0000 7f93a1f7da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:30:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:06.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:06.420 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:06.418+0000 7f93a1f7da40 -1 Falling back to public interface 2026-02-20T16:30:07.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:07.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:07.197 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:07.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:07.436 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:07.434+0000 7f93a1f7da40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:30:08.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:08.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:08.286 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:08.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:30:08.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:08.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:08.442 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3264635148,v1:127.0.0.1:6801/3264635148] [v2:127.0.0.1:6802/3264635148,v1:127.0.0.1:6803/3264635148] exists,up 1c593b15-4536-4223-96fd-b2bd8400d910 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:145: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:08.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:08.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-20T16:30:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:30:08.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:30:08.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11e9ef8c-431a-47b7-839c-7a5c4e75af61 2026-02-20T16:30:08.448 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 11e9ef8c-431a-47b7-839c-7a5c4e75af61 2026-02-20T16:30:08.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 11e9ef8c-431a-47b7-839c-7a5c4e75af61' 2026-02-20T16:30:08.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:30:08.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQjJhpvIFkGxAAzuKSoa0wtQaPX1IUMthI5Q== 2026-02-20T16:30:08.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQjJhpvIFkGxAAzuKSoa0wtQaPX1IUMthI5Q=="}' 2026-02-20T16:30:08.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11e9ef8c-431a-47b7-839c-7a5c4e75af61 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:30:08.620 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:08.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:30:08.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQAQjJhpvIFkGxAAzuKSoa0wtQaPX1IUMthI5Q== --osd-uuid 11e9ef8c-431a-47b7-839c-7a5c4e75af61 2026-02-20T16:30:08.646 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:08.646+0000 7f6fd19c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:08.648 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:08.646+0000 7f6fd19c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:08.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:08.646+0000 7f6fd19c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:08.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:08.646+0000 7f6fd19c8a40 -1 bdev(0x5607ba460c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:30:08.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:08.646+0000 7f6fd19c8a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:30:09.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:30:09.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:30:09.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:30:09.338 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:30:09.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:30:09.532 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:30:09.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:30:09.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-20T16:30:09.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:30:09.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:30:09.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:30:09.557 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:09.554+0000 7f0ab2d31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:09.559 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:09.558+0000 7f0ab2d31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:09.560 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:09.558+0000 7f0ab2d31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:30:09.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:09.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:09.974+0000 7f0ab2d31a40 -1 Falling back to public interface 2026-02-20T16:30:10.608 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:10.606+0000 7f0ab2d31a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:30:10.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:10.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:10.858 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:10.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:10.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:10.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:30:11.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:12.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:12.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:12.019 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:12.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:30:12.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:12.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1763262708,v1:127.0.0.1:6811/1763262708] [v2:127.0.0.1:6812/1763262708,v1:127.0.0.1:6813/1763262708] exists,up 11e9ef8c-431a-47b7-839c-7a5c4e75af61 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_allow_repair_during_recovery: create_rbd_pool 2026-02-20T16:30:12.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:30:12.325 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:30:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:30:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:30:12.545 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:30:12.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:30:13.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:30:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_allow_repair_during_recovery: wait_for_clean 2026-02-20T16:30:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:13.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:13.843 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:13.843 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:13.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:13.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:13.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:13.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:13.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:13.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:13.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:13.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:13.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:14.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:14.108 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:30:14.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:14.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:14.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:14.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:30:14.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:30:14.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:30:14.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:14.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:14.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-20T16:30:14.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-20T16:30:14.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-20T16:30:14.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:14.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:30:14.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:14.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:30:14.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:14.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:30:14.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:30:14.261 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:30:14.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:14.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T16:30:14.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:15.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:15.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:15.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:30:15.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:15.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-20T16:30:15.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:15.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:15.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-20T16:30:15.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:15.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-20T16:30:15.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-20T16:30:15.589 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705666 2026-02-20T16:30:15.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:15.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-20T16:30:15.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:15.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:15.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:15.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:30:15.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:15.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:15.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:15.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:15.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:15.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:15.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:16.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:30:16.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:16.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:16.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:30:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:30:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:30:16.506 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:30:16.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:30:16.710 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:30:16.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:30:16.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:30:16.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:30:16.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-20T16:30:16.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-20T16:30:16.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:30:16.750 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-20T16:30:16.750 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-20T16:30:16.750 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:30:16.750 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:30:16.750 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:30:16.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:30:16.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:30:16.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:30:17.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:17.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:17.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T16:30:17.574 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:30:17.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:30:17.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:17.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:30:17.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:30:17.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:30:17.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:17.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:17.672 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:17.672 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:17.672 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:17.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:30:17.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:30:17.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:30:17.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:30:17.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:30:17.696 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:17.694+0000 7fcc8b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:17.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:17.694+0000 7fcc8b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:17.698 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:17.694+0000 7fcc8b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:30:17.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:17.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:17.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:18.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:18.070+0000 7fcc8b4b1a40 -1 Falling back to public interface 2026-02-20T16:30:18.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:18.630+0000 7fcc8b4b1a40 -1 osd.0 18 log_to_monitors true 2026-02-20T16:30:18.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:18.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:18.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:19.000 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:19.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:19.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:19.159 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6800/2536379470,v1:127.0.0.1:6801/2536379470] [v2:127.0.0.1:6802/2536379470,v1:127.0.0.1:6803/2536379470] exists,up 1c593b15-4536-4223-96fd-b2bd8400d910 2026-02-20T16:30:19.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:19.160 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:19.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:19.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:19.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:19.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:19.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:19.428 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:30:19.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:19.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:19.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-20T16:30:19.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-20T16:30:19.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-20T16:30:19.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:19.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:19.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-20T16:30:19.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-20T16:30:19.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705668' 2026-02-20T16:30:19.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:19.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-20T16:30:19.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:19.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:19.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-20T16:30:19.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:19.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-20T16:30:19.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-20T16:30:19.578 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280514 2026-02-20T16:30:19.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:19.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-20T16:30:19.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:20.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:20.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-20T16:30:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:21.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:30:21.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:22.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-20T16:30:22.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:22.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-20T16:30:22.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:22.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:22.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-20T16:30:22.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:22.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-20T16:30:22.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-20T16:30:22.063 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705668 2026-02-20T16:30:22.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:22.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-20T16:30:22.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:22.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:22.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:22.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:30:22.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:22.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:22.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:22.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:22.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:22.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:22.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:22.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:30:22.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:22.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:22.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:30:22.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-20T16:30:22.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:22.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:22.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:22.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:23.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:23.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-20T16:30:23.226 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:23.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:23.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:23.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:23.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:12.545145+0000 '>' 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:30:24.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:30:24.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:24.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:30:24.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:24.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:24.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:24.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:24.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:12.545145+0000 '>' 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:24.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:30:25.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:30:25.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:25.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:30:25.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:25.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:25.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:25.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:25.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:12.545145+0000 '>' 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:25.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:30:26.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:30:26.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:26.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:30:26.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:26.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:26.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:26.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:26.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:12.545145+0000 '>' 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:26.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:30:27.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:30:27.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:27.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:30:27.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:30:27.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:27.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:27.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:23.686341+0000 '>' 2026-02-20T16:30:12.545145+0000 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:30:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:30:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:30:28.433 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-20T16:30:28.433 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-20T16:30:28.433 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:30:28.433 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:28.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:28.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:30:28.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:30:28.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:30:28.540 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:28.538+0000 7f5a2650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:28.541 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:28.538+0000 7f5a2650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:28.542 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:28.538+0000 7f5a2650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:28.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:30:28.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:28.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:30:28.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:28.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:28.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:28.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:28.683 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:30:28.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:28.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:30:28.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:28.968 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:28.966+0000 7f5a2650ba40 -1 Falling back to public interface 2026-02-20T16:30:29.664 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:29.662+0000 7f5a2650ba40 -1 osd.1 27 log_to_monitors true 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:29.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [9,27) [v2:127.0.0.1:6810/2686642723,v1:127.0.0.1:6811/2686642723] [v2:127.0.0.1:6812/2686642723,v1:127.0.0.1:6813/2686642723] exists,up 11e9ef8c-431a-47b7-839c-7a5c4e75af61 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:29.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:30.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:30.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:30.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:30.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:30.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:30.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:30.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:30.274 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:30:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:30.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:30.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280517 2026-02-20T16:30:30.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280517 2026-02-20T16:30:30.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517' 2026-02-20T16:30:30.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:30.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:30.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-20T16:30:30.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-20T16:30:30.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517 1-128849018882' 2026-02-20T16:30:30.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:30.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280517 2026-02-20T16:30:30.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:30.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:30.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280517 2026-02-20T16:30:30.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:30.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280517 2026-02-20T16:30:30.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2026-02-20T16:30:30.424 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280517 2026-02-20T16:30:30.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:30.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280516 -lt 94489280517 2026-02-20T16:30:30.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:31.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:31.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280517 2026-02-20T16:30:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018882 2026-02-20T16:30:31.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:31.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018882 2026-02-20T16:30:31.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:31.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-20T16:30:31.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2026-02-20T16:30:31.760 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 128849018882 2026-02-20T16:30:31.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:31.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-20T16:30:31.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:31.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:31.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:32.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:30:32.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:32.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:32.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:32.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:32.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:32.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:32.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:32.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:30:32.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:32.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:32.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:32.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:30:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:30:32.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:30:32.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:32.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:32.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:30:32.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:30:32.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:30:32.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:30:32.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:30:32.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:30:32.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:32.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:30:32.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:30:32.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:32.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:30:32.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:30:32.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:30:32.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:30:32.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:32.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:32.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:30:32.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:30:32.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:30:32.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:30:32.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:30:32.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:32.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:32.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:32.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:30:32.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:30:32.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:30:32.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:30:32.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:30:32.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:30:32.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:32.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:30:32.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:30:32.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:30:32.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:30:32.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:30:32.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:30:32.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:30:32.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:32.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:32.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:30:32.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:30:32.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:30:32.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:30:32.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:30:32.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:32.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:32.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:30:32.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:30:32.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:30:32.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:30:32.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:30:32.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:505: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=testpool 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=testpool 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-20T16:30:32.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-20T16:30:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2026-02-20T16:30:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T16:30:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T16:30:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T16:30:32.924 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 d0869380-8932-49cb-ac1a-c2d64cf5198a 2026-02-20T16:30:33.024 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:33.052 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.050+0000 7fcb7539ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:33.054 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.050+0000 7fcb7539ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:33.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.054+0000 7fcb7539ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:33.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.054+0000 7fcb7539ba40 -1 bdev(0x560c49a18c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:30:33.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.054+0000 7fcb7539ba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:30:33.840 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:30:33.942 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:30:33.974 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.970+0000 7f8aeacb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:33.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.974+0000 7f8aeacb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:33.977 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:33.974+0000 7f8aeacb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:34.037 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:34.536 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:34.534+0000 7f8aeacb5a40 -1 Falling back to public interface 2026-02-20T16:30:35.128 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:35.545 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:35.572+0000 7f8aeacb5a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:30:36.219 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:36.381 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/245205975,v1:127.0.0.1:6801/245205975] [v2:127.0.0.1:6802/245205975,v1:127.0.0.1:6803/245205975] exists,up d0869380-8932-49cb-ac1a-c2d64cf5198a 2026-02-20T16:30:36.386 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 c72f26a3-75fe-4794-b003-b9bc7504b478 2026-02-20T16:30:36.561 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:36.591 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:36.620+0000 7f51e96c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:36.593 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:36.620+0000 7f51e96c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:36.593 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:36.620+0000 7f51e96c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:36.594 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:36.620+0000 7f51e96c6a40 -1 bdev(0x56393b8a2c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:30:36.594 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:36.624+0000 7f51e96c6a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:30:37.271 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:30:37.471 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:30:37.501 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:37.528+0000 7fa3f0f6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:37.503 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:37.532+0000 7fa3f0f6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:37.504 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:37.532+0000 7fa3f0f6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:37.633 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:37.944 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:37.972+0000 7fa3f0f6fa40 -1 Falling back to public interface 2026-02-20T16:30:38.476 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:38.504+0000 7fa3f0f6fa40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:30:38.788 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:39.945 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:40.100 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/596571349,v1:127.0.0.1:6811/596571349] [v2:127.0.0.1:6812/596571349,v1:127.0.0.1:6813/596571349] exists,up c72f26a3-75fe-4794-b003-b9bc7504b478 2026-02-20T16:30:40.105 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 074c7c33-2530-48bb-9e1c-831c7ec0363a 2026-02-20T16:30:40.282 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:40.312 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:40.340+0000 7f1e85ef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:40.314 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:40.340+0000 7f1e85ef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:40.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:40.344+0000 7f1e85ef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:40.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:40.344+0000 7f1e85ef2a40 -1 bdev(0x562c447f4c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:30:40.316 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:40.344+0000 7f1e85ef2a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:30:40.984 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:30:41.195 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:30:41.225 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:41.252+0000 7f30f4056a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:41.227 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:41.256+0000 7f30f4056a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:41.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:41.256+0000 7f30f4056a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:41.351 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:30:41.564 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:41.596+0000 7f30f4056a40 -1 Falling back to public interface 2026-02-20T16:30:42.214 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:42.244+0000 7f30f4056a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:30:42.513 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:30:43.670 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:30:43.827 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2417127417,v1:127.0.0.1:6819/2417127417] [v2:127.0.0.1:6820/2417127417,v1:127.0.0.1:6821/2417127417] exists,up 074c7c33-2530-48bb-9e1c-831c7ec0363a 2026-02-20T16:30:44.072 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:30:45.579 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T16:30:48.063 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672967 2026-02-20T16:30:48.233 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T16:30:50.096 INFO:tasks.workunit.client.0.trial013.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2026-02-20T16:30:50.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:506: TEST_auto_repair_bluestore_basic: local poolid=1 2026-02-20T16:30:50.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:507: TEST_auto_repair_bluestore_basic: local poolname=testpool 2026-02-20T16:30:50.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:509: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2026-02-20T16:30:50.293 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:30:50.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-20T16:30:50.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:50.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:50.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:50.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:50.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:50.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:50.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:50.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:50.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:50.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:50.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:50.561 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:50.561 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:30:50.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:50.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:50.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:50.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-20T16:30:50.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-20T16:30:50.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-20T16:30:50.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:50.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:50.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672974 2026-02-20T16:30:50.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672974 2026-02-20T16:30:50.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2026-02-20T16:30:50.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:50.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:30:50.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-20T16:30:50.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-20T16:30:50.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-60129542154' 2026-02-20T16:30:50.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:50.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-20T16:30:50.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:50.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:50.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-20T16:30:50.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:50.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-20T16:30:50.782 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836496 2026-02-20T16:30:50.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-20T16:30:50.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:50.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-20T16:30:50.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:51.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:51.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:52.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836496 2026-02-20T16:30:52.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:52.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672974 2026-02-20T16:30:52.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:52.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:52.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672974 2026-02-20T16:30:52.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:52.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672974 2026-02-20T16:30:52.113 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672974 2026-02-20T16:30:52.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2026-02-20T16:30:52.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:52.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672974 -lt 42949672974 2026-02-20T16:30:52.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:52.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-20T16:30:52.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:52.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:30:52.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-20T16:30:52.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:52.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-20T16:30:52.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-20T16:30:52.281 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542154 2026-02-20T16:30:52.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:30:52.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-20T16:30:52.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:52.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:52.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:52.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:52.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:52.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:52.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:30:52.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:52.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:52.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:53.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:30:53.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:53.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:53.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2026-02-20T16:30:53.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:514: TEST_auto_repair_bluestore_basic: echo ABCDEF 2026-02-20T16:30:53.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:30:53.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-20T16:30:53.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:30:53.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:30:53.021 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:30:53.021 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:30:53.021 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:30:53.021 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:30:53.021 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:30:53.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:30:53.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:30:53.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:30:53.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:30:53.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:30:53.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:30:53.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:30:53.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:30:53.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:30:53.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:53.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T16:30:53.830 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:30:53.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:30:53.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:30:53.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:30:53.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:30:53.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:30:53.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:30:53.941 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:30:53.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:30:53.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:30:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:30:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:30:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:30:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:30:53.961 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:53.992+0000 7f9b68c9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:53.962 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:53.996+0000 7f9b68c9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:53.963 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:53.996+0000 7f9b68c9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:30:54.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:30:54.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:30:54.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:54.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:30:54.392 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:54.424+0000 7f9b68c9ea40 -1 Falling back to public interface 2026-02-20T16:30:54.945 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:30:54.976+0000 7f9b68c9ea40 -1 osd.0 21 log_to_monitors true 2026-02-20T16:30:55.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:30:55.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:30:55.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:30:55.262 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:55.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:30:55.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:30:55.422 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6800/61390963,v1:127.0.0.1:6801/61390963] [v2:127.0.0.1:6802/61390963,v1:127.0.0.1:6803/61390963] exists,up d0869380-8932-49cb-ac1a-c2d64cf5198a 2026-02-20T16:30:55.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:30:55.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:30:55.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:30:55.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:30:55.425 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:30:55.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:30:55.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:30:55.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:30:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:30:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:30:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:30:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:30:55.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:30:55.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:55.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:30:55.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-20T16:30:55.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-20T16:30:55.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-20T16:30:55.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:55.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:30:55.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672980 2026-02-20T16:30:55.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672980 2026-02-20T16:30:55.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672980' 2026-02-20T16:30:55.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:30:55.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:30:55.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542160 2026-02-20T16:30:55.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542160 2026-02-20T16:30:55.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672980 2-60129542160' 2026-02-20T16:30:55.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:55.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-20T16:30:55.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:55.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:30:55.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-20T16:30:55.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:55.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-20T16:30:55.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-20T16:30:55.919 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 103079215106 2026-02-20T16:30:55.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:56.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-20T16:30:56.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:30:57.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:30:57.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:30:57.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215106 2026-02-20T16:30:57.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:57.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672980 2026-02-20T16:30:57.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:57.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:30:57.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672980 2026-02-20T16:30:57.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:57.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672980 2026-02-20T16:30:57.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2026-02-20T16:30:57.242 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672980 2026-02-20T16:30:57.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:30:57.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672981 -lt 42949672980 2026-02-20T16:30:57.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:30:57.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542160 2026-02-20T16:30:57.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:30:57.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:30:57.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542160 2026-02-20T16:30:57.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:30:57.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542160 2026-02-20T16:30:57.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2026-02-20T16:30:57.410 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542160 2026-02-20T16:30:57.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:30:57.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542161 -lt 60129542160 2026-02-20T16:30:57.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:30:57.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:57.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:57.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:30:57.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:30:57.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:30:57.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:30:57.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:30:57.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:30:57.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:30:58.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:30:58.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:30:58.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:30:58.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-20T16:30:58.163 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T16:30:58.163 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T16:30:58.163 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:30:58.171 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T16:30:58.171 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-20T16:30:58.171 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:30:58.178 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T16:30:58.178 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-20T16:30:58.178 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:30:58.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2026-02-20T16:30:58.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:30:58.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:30:58.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-20T16:30:58.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:30:58.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: local pgid=1.0 2026-02-20T16:30:58.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2026-02-20T16:30:58.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:30:58.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:30:58.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:30:58.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:30:58.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: local primary=1 2026-02-20T16:30:58.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2026-02-20T16:30:58.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:30:58.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:58.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:58.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:30:58.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2026-02-20T16:30:44.104066+0000 2026-02-20T16:30:58.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2026-02-20T16:30:58.723 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:30:58.723 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:30:58.723 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:30:58.723 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T11:42:48.757536+0000" 2026-02-20T16:30:58.723 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2026-02-20T16:30:44.104066+0000 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:30:44.104066+0000 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:30:58.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:30:58.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:58.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:58.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:30:58.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:44.104066+0000 '>' 2026-02-20T16:30:44.104066+0000 2026-02-20T16:30:58.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:30:59.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:30:59.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:30:59.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:30:59.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:30:59.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:30:59.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:30:59.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:00.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:44.104066+0000 '>' 2026-02-20T16:30:44.104066+0000 2026-02-20T16:31:00.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:31:01.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:31:01.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:31:01.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:31:01.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:31:01.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:31:01.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:31:01.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:01.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:30:59.667287+0000 '>' 2026-02-20T16:30:44.104066+0000 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:01.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:01.205 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:01.205 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:01.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:01.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:01.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:01.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:01.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:01.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:01.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:01.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:01.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:01.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:01.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215113 2026-02-20T16:31:01.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215113 2026-02-20T16:31:01.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113' 2026-02-20T16:31:01.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:01.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:01.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672986 2026-02-20T16:31:01.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672986 2026-02-20T16:31:01.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113 1-42949672986' 2026-02-20T16:31:01.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:01.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:01.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542167 2026-02-20T16:31:01.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542167 2026-02-20T16:31:01.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113 1-42949672986 2-60129542167' 2026-02-20T16:31:01.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:01.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215113 2026-02-20T16:31:01.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:01.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:01.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215113 2026-02-20T16:31:01.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:01.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215113 2026-02-20T16:31:01.692 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 103079215113 2026-02-20T16:31:01.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215113' 2026-02-20T16:31:01.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:01.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215112 -lt 103079215113 2026-02-20T16:31:01.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:02.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:02.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:03.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215112 -lt 103079215113 2026-02-20T16:31:03.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:04.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:31:04.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:04.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215115 -lt 103079215113 2026-02-20T16:31:04.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:04.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672986 2026-02-20T16:31:04.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:04.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:04.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672986 2026-02-20T16:31:04.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:04.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672986 2026-02-20T16:31:04.181 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672986 2026-02-20T16:31:04.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2026-02-20T16:31:04.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:04.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672988 -lt 42949672986 2026-02-20T16:31:04.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:04.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542167 2026-02-20T16:31:04.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:04.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:04.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542167 2026-02-20T16:31:04.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:04.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542167 2026-02-20T16:31:04.356 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542167 2026-02-20T16:31:04.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2026-02-20T16:31:04.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:04.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542168 -lt 60129542167 2026-02-20T16:31:04.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:04.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:04.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:04.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:04.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:04.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:04.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:04.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:04.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:04.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:04.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:04.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:04.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:04.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:04.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:05.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:05.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:05.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:05.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:532: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2026-02-20T16:31:05.223 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:31:05.224 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-20T16:30:59.669479+0000 20'1 25:68 [1,0] 1 [1,0] 1 20'1 2026-02-20T16:30:59.667287+0000 20'1 2026-02-20T16:30:59.667287+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:30:59.667287+0000 1 0 2026-02-20T16:31:05.224 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:31:05.224 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:31:05.224 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:31:05.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-20T16:31:05.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:31:05.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:31:05.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:31:05.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:05.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:31:05.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:05.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:05.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:31:05.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:05.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:31:05.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:31:05.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:05.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:05.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:31:05.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:05.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:05.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:05.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:31:05.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:05.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:05.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:31:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-20T16:31:05.941 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:31:05.941 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:31:06.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:31:06.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:06.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:06.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:31:06.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:06.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:06.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:31:06.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:31:06.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:31:06.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:31:06.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:06.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:06.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:06.067 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:06.100+0000 7f1073021a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:06.068 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:06.100+0000 7f1073021a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:06.069 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:06.100+0000 7f1073021a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:06.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:31:06.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:06.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:06.480 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:06.512+0000 7f1073021a40 -1 Falling back to public interface 2026-02-20T16:31:07.157 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:07.188+0000 7f1073021a40 -1 osd.0 25 log_to_monitors true 2026-02-20T16:31:07.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:07.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:07.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:07.372 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:07.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:07.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:07.530 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6800/3466977509,v1:127.0.0.1:6801/3466977509] [v2:127.0.0.1:6802/3466977509,v1:127.0.0.1:6803/3466977509] exists,up d0869380-8932-49cb-ac1a-c2d64cf5198a 2026-02-20T16:31:07.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:07.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:07.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:07.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:31:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:07.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:07.532 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:07.532 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:07.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:07.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:07.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:07.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:07.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-20T16:31:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-20T16:31:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-20T16:31:07.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:07.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672994 2026-02-20T16:31:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672994 2026-02-20T16:31:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672994' 2026-02-20T16:31:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:07.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:08.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542174 2026-02-20T16:31:08.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542174 2026-02-20T16:31:08.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672994 2-60129542174' 2026-02-20T16:31:08.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:08.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-20T16:31:08.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:08.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:08.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-20T16:31:08.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:08.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-20T16:31:08.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-20T16:31:08.031 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 124554051586 2026-02-20T16:31:08.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:08.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215117 -lt 124554051586 2026-02-20T16:31:08.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:09.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:09.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:09.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051587 -lt 124554051586 2026-02-20T16:31:09.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:09.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672994 2026-02-20T16:31:09.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:09.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:09.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672994 2026-02-20T16:31:09.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:09.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672994 2026-02-20T16:31:09.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2026-02-20T16:31:09.353 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672994 2026-02-20T16:31:09.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:09.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672995 -lt 42949672994 2026-02-20T16:31:09.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542174 2026-02-20T16:31:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:09.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:09.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542174 2026-02-20T16:31:09.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:09.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542174 2026-02-20T16:31:09.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2026-02-20T16:31:09.513 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542174 2026-02-20T16:31:09.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:09.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542175 -lt 60129542174 2026-02-20T16:31:09.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:09.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:09.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:09.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:10.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:10.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:10.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:10.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:10.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:10.208 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:31:10.209 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:10.209 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:10.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:31:10.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:10.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:31:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:31:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:31:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:10.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:31:11.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:31:11.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:11.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:31:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:11.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:31:11.016 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:31:11.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:31:11.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:31:11.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:31:11.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:11.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:11.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:11.031 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:11.064+0000 7f2b4552ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:11.032 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:11.064+0000 7f2b4552ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:11.033 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:11.064+0000 7f2b4552ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:11.175 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:11.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:11.476 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:11.508+0000 7f2b4552ba40 -1 Falling back to public interface 2026-02-20T16:31:12.047 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:12.080+0000 7f2b4552ba40 -1 osd.0 30 log_to_monitors true 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:12.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:12.494 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6800/2604015816,v1:127.0.0.1:6801/2604015816] [v2:127.0.0.1:6802/2604015816,v1:127.0.0.1:6803/2604015816] exists,up d0869380-8932-49cb-ac1a-c2d64cf5198a 2026-02-20T16:31:12.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:12.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:12.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:12.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:12.495 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:12.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:12.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:12.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:12.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:12.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:12.756 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:12.756 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:12.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:12.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-20T16:31:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-20T16:31:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-20T16:31:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:12.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:12.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673000 2026-02-20T16:31:12.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673000 2026-02-20T16:31:12.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949673000' 2026-02-20T16:31:12.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:12.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542180 2026-02-20T16:31:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542180 2026-02-20T16:31:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949673000 2-60129542180' 2026-02-20T16:31:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:12.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-20T16:31:12.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:12.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:12.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-20T16:31:12.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:12.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-20T16:31:12.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-20T16:31:12.979 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 146028888066 2026-02-20T16:31:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:13.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051589 -lt 146028888066 2026-02-20T16:31:13.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:14.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:14.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:14.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888067 -lt 146028888066 2026-02-20T16:31:14.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:14.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673000 2026-02-20T16:31:14.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:14.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:14.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673000 2026-02-20T16:31:14.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:14.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673000 2026-02-20T16:31:14.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2026-02-20T16:31:14.308 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949673000 2026-02-20T16:31:14.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:14.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673000 -lt 42949673000 2026-02-20T16:31:14.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:14.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542180 2026-02-20T16:31:14.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:14.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542180 2026-02-20T16:31:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:14.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542180 2026-02-20T16:31:14.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2026-02-20T16:31:14.476 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542180 2026-02-20T16:31:14.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:14.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542180 -lt 60129542180 2026-02-20T16:31:14.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:14.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:14.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:14.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:14.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:14.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:14.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:14.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:14.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:14.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:14.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:15.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:15.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:15.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:15.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:537: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:31:15.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:538: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:31:15.200 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:30:59.664+0000 7fa3ce6ec640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:31:15.200 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:30:59.664+0000 7fa3ce6ec640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:31:15.200 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:30:59.664+0000 7fa3ce6ec640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-20T16:31:15.200 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:30:59.664+0000 7fa3ce6ec640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-20T16:31:15.201 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:30:59.664+0000 7fa3ce6ec640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:15.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:15.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:15.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:15.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:15.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:31:15.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:31:15.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:31:15.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:31:15.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:31:15.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:31:15.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:31:15.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:31:15.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:31:15.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:31:15.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:31:15.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:31:15.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:31:15.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:31:15.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:31:15.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:31:15.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:31:15.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:15.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:15.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:15.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:15.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:15.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:15.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:31:15.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:31:15.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:31:15.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:31:15.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:31:15.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:31:15.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:31:15.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:31:15.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:31:15.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:31:15.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:31:15.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:31:15.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:31:15.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:31:15.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:31:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:31:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:31:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:31:15.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:31:15.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:31:15.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: local poolname=testpool 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:602: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:31:15.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:31:15.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:31:15.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:31:15.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:31:15.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:15.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:15.370 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:15.370 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.370 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:15.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:31:15.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:31:15.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:31:15.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:31:15.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:31:15.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:31:15.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:31:15.402 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:31:15.402 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:31:15.402 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:31:15.402 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:31:15.403 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.403 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.403 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:31:15.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:31:15.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:31:15.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:31:15.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:31:15.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:31:15.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:31:15.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.458 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:31:15.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:31:15.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:31:15.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:603: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:31:15.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:31:15.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:15.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:31:15.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:31:15.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:606: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:15.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: seq 0 2 2026-02-20T16:31:15.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:15.633 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:15.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:15.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:15.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:31:15.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:15.636 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:15.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45' 2026-02-20T16:31:15.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:31:15.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTjJhpNEyUKBAApjh9KlfzwOXZc2FJQgvZAg== 2026-02-20T16:31:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTjJhpNEyUKBAApjh9KlfzwOXZc2FJQgvZAg=="}' 2026-02-20T16:31:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:31:15.743 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:31:15.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:31:15.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBTjJhpNEyUKBAApjh9KlfzwOXZc2FJQgvZAg== --osd-uuid 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:15.769 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:15.800+0000 7eff0c5cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:15.770 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:15.804+0000 7eff0c5cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:15.771 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:15.804+0000 7eff0c5cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:15.772 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:15.804+0000 7eff0c5cfa40 -1 bdev(0x5646d9cbec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:31:15.772 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:15.804+0000 7eff0c5cfa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:31:16.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:31:16.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:31:16.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:31:16.580 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:31:16.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:31:16.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:31:16.684 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:31:16.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:16.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:31:16.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:31:16.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:31:16.709 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:16.740+0000 7fa0dc766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:16.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:16.744+0000 7fa0dc766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:16.713 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:16.744+0000 7fa0dc766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:16.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:31:16.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:16.780 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:31:16.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:16.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:16.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:17.212 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:17.244+0000 7fa0dc766a40 -1 Falling back to public interface 2026-02-20T16:31:17.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:17.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:17.872 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:31:17.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:17.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:17.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:17.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:18.100 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:18.132+0000 7fa0dc766a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:31:18.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:18.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:18.962 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:31:18.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:31:18.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:18.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:19.119 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1093990123,v1:127.0.0.1:6801/1093990123] [v2:127.0.0.1:6802/1093990123,v1:127.0.0.1:6803/1093990123] exists,up 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:19.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:19.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:19.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:31:19.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:31:19.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:31:19.125 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:31:19.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 cd980eee-c89c-4efc-8f38-2ae86bc3479d' 2026-02-20T16:31:19.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:31:19.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXjJhp0QovChAAvquTVXSdyOEMEp/JfEsDvA== 2026-02-20T16:31:19.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXjJhp0QovChAAvquTVXSdyOEMEp/JfEsDvA=="}' 2026-02-20T16:31:19.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cd980eee-c89c-4efc-8f38-2ae86bc3479d -i td/osd-scrub-repair/1/new.json 2026-02-20T16:31:19.299 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:31:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:31:19.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBXjJhp0QovChAAvquTVXSdyOEMEp/JfEsDvA== --osd-uuid cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:31:19.334 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:19.364+0000 7fab17ed5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:19.335 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:19.368+0000 7fab17ed5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:19.336 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:19.368+0000 7fab17ed5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:19.337 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:19.368+0000 7fab17ed5a40 -1 bdev(0x559971696c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:31:19.337 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:19.368+0000 7fab17ed5a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:31:19.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:31:19.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:31:19.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:31:19.999 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:31:19.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:31:20.205 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:31:20.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:31:20.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:20.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:31:20.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:31:20.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:31:20.231 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:20.264+0000 7f9bfa707a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:20.232 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:20.264+0000 7f9bfa707a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:20.233 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:20.264+0000 7f9bfa707a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:20.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:20.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:20.362 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:31:20.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:20.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:31:20.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:20.640 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:20.672+0000 7f9bfa707a40 -1 Falling back to public interface 2026-02-20T16:31:21.277 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:21.308+0000 7f9bfa707a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:31:21.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:21.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:21.519 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:31:21.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:21.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:21.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:31:21.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:22.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:31:22.840 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4022997994,v1:127.0.0.1:6811/4022997994] [v2:127.0.0.1:6812/4022997994,v1:127.0.0.1:6813/4022997994] exists,up cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:31:22.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:22.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:22.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:22.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:22.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:22.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:31:22.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:31:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e487830-1b5a-4ef2-89ee-01225a2446f1 2026-02-20T16:31:22.847 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 5e487830-1b5a-4ef2-89ee-01225a2446f1 2026-02-20T16:31:22.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5e487830-1b5a-4ef2-89ee-01225a2446f1' 2026-02-20T16:31:22.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:31:22.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBajJhpGWQJNRAAcqqKe97GLgUH19HHsVWE8g== 2026-02-20T16:31:22.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBajJhpGWQJNRAAcqqKe97GLgUH19HHsVWE8g=="}' 2026-02-20T16:31:22.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e487830-1b5a-4ef2-89ee-01225a2446f1 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:31:23.022 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:31:23.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:31:23.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBajJhpGWQJNRAAcqqKe97GLgUH19HHsVWE8g== --osd-uuid 5e487830-1b5a-4ef2-89ee-01225a2446f1 2026-02-20T16:31:23.049 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.080+0000 7f5d27144a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:23.051 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.084+0000 7f5d27144a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:23.052 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.084+0000 7f5d27144a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:23.053 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.084+0000 7f5d27144a40 -1 bdev(0x5563f3006c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:31:23.053 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.084+0000 7f5d27144a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:31:23.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:31:23.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:31:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:31:23.726 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:31:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:31:23.930 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:31:23.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:31:23.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:23.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:31:23.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:31:23.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:31:23.955 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.988+0000 7f0f6a33fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:23.956 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.988+0000 7f0f6a33fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:23.958 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:23.988+0000 7f0f6a33fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:24.087 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:31:24.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:24.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:24.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:31:24.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:24.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:24.408+0000 7f0f6a33fa40 -1 Falling back to public interface 2026-02-20T16:31:25.080 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:25.112+0000 7f0f6a33fa40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:31:25.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:25.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:25.247 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:31:25.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:25.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:25.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:31:25.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:26.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:26.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:26.408 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:31:26.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:31:26.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:26.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:31:26.570 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1063736259,v1:127.0.0.1:6819/1063736259] [v2:127.0.0.1:6820/1063736259,v1:127.0.0.1:6821/1063736259] exists,up 5e487830-1b5a-4ef2-89ee-01225a2446f1 2026-02-20T16:31:26.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:26.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:26.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:26.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2026-02-20T16:31:26.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-20T16:31:26.815 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:31:26.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:31:27.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2026-02-20T16:31:27.991 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:31:28.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-20T16:31:28.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:28.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:28.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:28.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:28.002 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:28.002 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:28.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:28.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:28.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:28.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:28.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:28.275 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:28.275 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:28.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:28.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:28.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:28.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:31:28.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:31:28.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:31:28.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:28.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:28.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:31:28.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:31:28.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-20T16:31:28.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:28.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:31:28.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:28.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:28.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:31:28.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:28.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:31:28.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:31:28.496 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:31:28.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:28.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:31:28.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:29.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:29.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:29.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:31:29.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:30.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:31:30.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:30.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:31:30.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:30.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:31:30.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:30.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:30.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:31:30.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:30.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:31:30.988 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:31:30.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:31:30.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:31.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:31:31.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:31.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-20T16:31:31.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:31.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:31.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-20T16:31:31.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:31.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-20T16:31:31.155 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574850 2026-02-20T16:31:31.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-20T16:31:31.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:31.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-20T16:31:31.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:31.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:31.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:31.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:31.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:31.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:31.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:31.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:31.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:31.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:31.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:31.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:31.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:31.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:31.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:616: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed: echo ABCDEF 2026-02-20T16:31:31.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: seq 1 10 2026-02-20T16:31:31.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:31.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:31.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:31.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:31.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:31.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:31.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:32.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:32.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:32.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:32.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:32.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:32.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:32.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-20T16:31:32.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:32.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:31:32.105 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:32.105 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:32.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:31:32.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:32.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:31:32.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:32.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:32.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:32.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:31:32.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:32.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:32.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:31:32.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:32.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:32.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-20T16:31:32.926 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T16:31:33.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:33.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:33.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:33.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:33.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:33.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:33.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:31:33.021 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:33.038 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:33.067+0000 7fb0e5e71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:33.039 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:33.071+0000 7fb0e5e71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:33.040 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:33.071+0000 7fb0e5e71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:33.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:31:33.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:33.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:33.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:33.424 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:33.455+0000 7fb0e5e71a40 -1 Falling back to public interface 2026-02-20T16:31:34.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:34.087+0000 7fb0e5e71a40 -1 osd.0 19 log_to_monitors true 2026-02-20T16:31:34.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:34.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:34.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:34.339 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:34.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:34.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:34.497 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/4045044931,v1:127.0.0.1:6801/4045044931] [v2:127.0.0.1:6802/4045044931,v1:127.0.0.1:6803/4045044931] exists,up 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:34.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:34.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:34.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:34.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:34.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:34.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:34.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:34.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:31:34.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:31:34.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:31:34.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:34.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:34.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:31:34.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:31:34.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705669' 2026-02-20T16:31:34.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:34.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705669 2-55834574852' 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:31:35.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:35.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:35.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:31:35.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:35.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:31:35.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:31:35.004 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247810 2026-02-20T16:31:35.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:35.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 98784247810 2026-02-20T16:31:35.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:36.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:36.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:36.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-20T16:31:36.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:36.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:31:36.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:36.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:31:36.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:36.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:31:36.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:31:36.326 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705669 2026-02-20T16:31:36.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:36.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:31:36.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:36.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-20T16:31:36.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:36.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:36.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-20T16:31:36.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:36.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-20T16:31:36.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-20T16:31:36.498 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574852 2026-02-20T16:31:36.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:36.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-20T16:31:36.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:36.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:36.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:36.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:36.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:36.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:36.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:36.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:36.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:36.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:36.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:37.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:37.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:37.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:37.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:37.202 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:31:37.203 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:37.203 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:37.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:31:37.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:37.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:31:37.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:31:37.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:37.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:37.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:37.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:37.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:37.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:37.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-20T16:31:38.017 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:104778fc:::obj2:head# 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:38.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:31:38.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:31:38.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:38.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:38.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:38.141 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:38.171+0000 7f75c6fafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:38.143 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:38.175+0000 7f75c6fafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:38.143 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:38.175+0000 7f75c6fafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:38.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:31:38.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:38.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:38.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:38.587 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:38.619+0000 7f75c6fafa40 -1 Falling back to public interface 2026-02-20T16:31:39.139 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:39.171+0000 7f75c6fafa40 -1 osd.0 25 log_to_monitors true 2026-02-20T16:31:39.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:39.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:39.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:39.444 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:39.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:39.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [23,25) [v2:127.0.0.1:6800/3627830113,v1:127.0.0.1:6801/3627830113] [v2:127.0.0.1:6802/3627830113,v1:127.0.0.1:6803/3627830113] exists,up 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:39.610 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:39.611 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:39.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:39.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:39.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:39.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:39.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:39.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-20T16:31:39.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-20T16:31:39.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-20T16:31:39.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:39.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:40.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-20T16:31:40.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-20T16:31:40.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-38654705671' 2026-02-20T16:31:40.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:40.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:40.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-20T16:31:40.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-20T16:31:40.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-38654705671 2-55834574854' 2026-02-20T16:31:40.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:40.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-20T16:31:40.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:40.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:40.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-20T16:31:40.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:40.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-20T16:31:40.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-20T16:31:40.092 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 124554051586 2026-02-20T16:31:40.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:40.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 124554051586 2026-02-20T16:31:40.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:41.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:41.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:41.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 124554051586 2026-02-20T16:31:41.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:42.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:31:42.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:42.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-20T16:31:42.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:42.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-20T16:31:42.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:42.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:42.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-20T16:31:42.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:42.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-20T16:31:42.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-20T16:31:42.581 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705671 2026-02-20T16:31:42.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:42.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705671 -lt 38654705671 2026-02-20T16:31:42.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:42.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574854 2026-02-20T16:31:42.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:42.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574854 2026-02-20T16:31:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:42.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574854 2026-02-20T16:31:42.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-20T16:31:42.749 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574854 2026-02-20T16:31:42.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:42.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574854 2026-02-20T16:31:42.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:42.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:42.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:43.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:43.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:31:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:43.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:31:43.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T16:31:43.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:31:43.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T16:31:43.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:31:43.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:31:43.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:31:43.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:43.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:43.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:31:43.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:43.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:43.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:31:43.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:31:43.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:31:43.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:31:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:31:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:31:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:31:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:31:44.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:31:44.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:31:44.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:44.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:44.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:44.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:44.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:44.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:44.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:44.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:44.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:44.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:31:44.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:31:44.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:44.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:44.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:44.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:44.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:45.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:45.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:31:45.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-20T16:31:45.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:45.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:45.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:45.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:31:45.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:31:45.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:31:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:45.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:45.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:45.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:45.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:45.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:45.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:45.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:45.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:31:45.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:45.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:31:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:31:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:45.462 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:45.491+0000 7f87c5f78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:45.463 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:45.495+0000 7f87c5f78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:45.464 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:45.495+0000 7f87c5f78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:45.606 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:45.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:45.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:31:45.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:45.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:45.895+0000 7f87c5f78a40 -1 Falling back to public interface 2026-02-20T16:31:46.484 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:46.515+0000 7f87c5f78a40 -1 osd.1 30 log_to_monitors true 2026-02-20T16:31:46.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:46.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:46.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:46.766 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:46.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:46.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [9,30) [v2:127.0.0.1:6810/271499408,v1:127.0.0.1:6811/271499408] [v2:127.0.0.1:6812/271499408,v1:127.0.0.1:6813/271499408] exists,up cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:31:46.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:46.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:46.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:46.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:46.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:46.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:46.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:46.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:46.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:47.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:47.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:47.185 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:47.185 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:47.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:47.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:47.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:47.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051588 2026-02-20T16:31:47.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051588 2026-02-20T16:31:47.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051588' 2026-02-20T16:31:47.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:47.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:47.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-20T16:31:47.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-20T16:31:47.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051588 1-146028888066' 2026-02-20T16:31:47.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:47.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:47.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-20T16:31:47.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-20T16:31:47.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051588 1-146028888066 2-55834574857' 2026-02-20T16:31:47.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:47.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051588 2026-02-20T16:31:47.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:47.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:47.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051588 2026-02-20T16:31:47.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:47.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051588 2026-02-20T16:31:47.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051588' 2026-02-20T16:31:47.405 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 124554051588 2026-02-20T16:31:47.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:47.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051587 -lt 124554051588 2026-02-20T16:31:47.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:48.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:48.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:48.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051588 2026-02-20T16:31:48.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:48.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888066 2026-02-20T16:31:48.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:48.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:48.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888066 2026-02-20T16:31:48.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:48.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-20T16:31:48.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2026-02-20T16:31:48.741 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 146028888066 2026-02-20T16:31:48.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:48.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-20T16:31:48.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:48.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-20T16:31:48.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:48.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:48.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-20T16:31:48.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:48.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-20T16:31:48.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-20T16:31:48.908 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574857 2026-02-20T16:31:48.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:49.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-20T16:31:49.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:49.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:49.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:49.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:49.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:49.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:49.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:49.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:49.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:49.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:31:49.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-20T16:31:49.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-20T16:31:49.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:31:49.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: local pgid=1.0 2026-02-20T16:31:49.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2026-02-20T16:31:49.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:49.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:31:49.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:31:49.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:49.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: local primary=1 2026-02-20T16:31:49.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2026-02-20T16:31:49.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:31:49.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:31:49.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:31:49.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:50.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:50.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2026-02-20T16:31:50.141 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:31:50.141 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:31:50.141 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:31:50.141 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T11:43:40.175441+0000" 2026-02-20T16:31:50.141 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:31:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:31:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:31:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:31:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:50.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:26.847053+0000 '>' 2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:50.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:31:51.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:31:51.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:31:51.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:31:51.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:31:51.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:31:51.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:31:51.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:51.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:26.847053+0000 '>' 2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:51.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:31:52.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:31:52.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:31:52.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:31:52.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:31:52.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:31:52.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:31:52.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:31:52.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:50.621635+0000 '>' 2026-02-20T16:31:26.847053+0000 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:31:52.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:31:52.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:31:52.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:31:52.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:31:52.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:31:52.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:31:52.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:31:52.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:52.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:52.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:52.889 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:52.889 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:52.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:52.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:52.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:52.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051590 2026-02-20T16:31:52.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051590 2026-02-20T16:31:52.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590' 2026-02-20T16:31:52.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:52.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:53.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888068 2026-02-20T16:31:53.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888068 2026-02-20T16:31:53.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590 1-146028888068' 2026-02-20T16:31:53.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:53.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:53.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-20T16:31:53.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-20T16:31:53.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590 1-146028888068 2-55834574859' 2026-02-20T16:31:53.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:53.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051590 2026-02-20T16:31:53.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:53.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:53.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051590 2026-02-20T16:31:53.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:53.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051590 2026-02-20T16:31:53.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2026-02-20T16:31:53.113 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 124554051590 2026-02-20T16:31:53.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:53.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051589 -lt 124554051590 2026-02-20T16:31:53.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:54.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:54.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:54.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051590 -lt 124554051590 2026-02-20T16:31:54.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:54.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888068 2026-02-20T16:31:54.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:54.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:54.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888068 2026-02-20T16:31:54.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:54.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888068 2026-02-20T16:31:54.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888068' 2026-02-20T16:31:54.449 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 146028888068 2026-02-20T16:31:54.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888068 -lt 146028888068 2026-02-20T16:31:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:54.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-20T16:31:54.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:54.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:54.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574859 2026-02-20T16:31:54.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:54.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574859 2026-02-20T16:31:54.617 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574859 2026-02-20T16:31:54.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-20T16:31:54.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:54.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-20T16:31:54.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:31:54.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:54.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:54.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:31:54.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:31:55.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:31:55.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:31:55.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:31:55.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:638: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:31:55.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:31:55.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:31:55.504 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:55.504 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:31:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:31:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:55.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:31:55.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051592 2026-02-20T16:31:55.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051592 2026-02-20T16:31:55.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051592' 2026-02-20T16:31:55.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:55.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:31:55.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888069 2026-02-20T16:31:55.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888069 2026-02-20T16:31:55.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051592 1-146028888069' 2026-02-20T16:31:55.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:31:55.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:31:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-20T16:31:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-20T16:31:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051592 1-146028888069 2-55834574860' 2026-02-20T16:31:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:55.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051592 2026-02-20T16:31:55.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:55.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:31:55.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051592 2026-02-20T16:31:55.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:55.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051592 2026-02-20T16:31:55.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2026-02-20T16:31:55.725 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 124554051592 2026-02-20T16:31:55.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:55.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051590 -lt 124554051592 2026-02-20T16:31:55.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:31:56.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:31:56.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:31:57.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051592 -lt 124554051592 2026-02-20T16:31:57.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:57.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888069 2026-02-20T16:31:57.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:57.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:31:57.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888069 2026-02-20T16:31:57.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:57.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888069 2026-02-20T16:31:57.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2026-02-20T16:31:57.059 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 146028888069 2026-02-20T16:31:57.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:31:57.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888069 -lt 146028888069 2026-02-20T16:31:57.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:31:57.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574860 2026-02-20T16:31:57.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:31:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:31:57.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574860 2026-02-20T16:31:57.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:31:57.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574860 2026-02-20T16:31:57.225 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574860 2026-02-20T16:31:57.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-20T16:31:57.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:31:57.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-20T16:31:57.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:31:57.391 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:31:57.391 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 15 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 20 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-20T16:31:57.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2026-02-20T16:31:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-20T16:31:57.550 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:31:57.550 INFO:tasks.workunit.client.0.trial013.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-20T16:31:51.638386+0000 19'10 36:109 [1,0] 1 [1,0] 1 19'10 2026-02-20T16:31:51.638345+0000 19'10 2026-02-20T16:31:51.638345+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:31:51.638345+0000 10 0 2026-02-20T16:31:57.550 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:31:57.550 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:31:57.551 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:31:57.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-20T16:31:57.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2026-02-20T16:31:57.723 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:31:57.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:31:57.733 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:31:57.734 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:31:57.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:31:57.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-20T16:31:57.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:31:58.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:31:58.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:31:58.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:31:58.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:58.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2026-02-20T16:31:58.429 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:31:58.429 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:31:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:31:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:31:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:31:58.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:31:58.521 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:31:58.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:31:58.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:31:58.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:31:58.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:31:58.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:31:58.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:31:58.540 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:58.571+0000 7f19f1169a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:58.541 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:58.571+0000 7f19f1169a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:58.542 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:58.575+0000 7f19f1169a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:31:58.679 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:31:58.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:58.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:31:58.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:31:58.883 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:58.915+0000 7f19f1169a40 -1 Falling back to public interface 2026-02-20T16:31:59.513 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:31:59.543+0000 7f19f1169a40 -1 osd.0 36 log_to_monitors true 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:31:59.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 40 up_thru 31 down_at 37 last_clean_interval [29,36) [v2:127.0.0.1:6800/2332166386,v1:127.0.0.1:6801/2332166386] [v2:127.0.0.1:6802/2332166386,v1:127.0.0.1:6803/2332166386] exists,up 650b7be1-243f-4c2d-b8ae-bc4f2a5fce45 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:00.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:00.009 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:00.010 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:00.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:00.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:00.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:00.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:00.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:00.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:00.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:00.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:00.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:00.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:00.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-20T16:32:00.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-20T16:32:00.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842' 2026-02-20T16:32:00.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:00.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:00.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888071 2026-02-20T16:32:00.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888071 2026-02-20T16:32:00.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-146028888071' 2026-02-20T16:32:00.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:00.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574862 2026-02-20T16:32:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574862 2026-02-20T16:32:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-146028888071 2-55834574862' 2026-02-20T16:32:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:00.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691842 2026-02-20T16:32:00.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:00.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:00.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691842 2026-02-20T16:32:00.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:00.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-20T16:32:00.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2026-02-20T16:32:00.494 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 171798691842 2026-02-20T16:32:00.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:00.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 171798691842 2026-02-20T16:32:00.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:01.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:01.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:01.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 171798691842 2026-02-20T16:32:01.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:02.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:32:02.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:02.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-20T16:32:02.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:02.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888071 2026-02-20T16:32:02.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:02.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:02.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888071 2026-02-20T16:32:02.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:02.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888071 2026-02-20T16:32:02.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888071' 2026-02-20T16:32:02.986 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 146028888071 2026-02-20T16:32:02.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:03.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888072 -lt 146028888071 2026-02-20T16:32:03.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:03.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574862 2026-02-20T16:32:03.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:03.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:03.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574862 2026-02-20T16:32:03.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:03.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574862 2026-02-20T16:32:03.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2026-02-20T16:32:03.149 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574862 2026-02-20T16:32:03.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:03.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574863 -lt 55834574862 2026-02-20T16:32:03.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:03.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:03.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:03.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:03.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:03.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:03.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:03.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:03.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:03.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:03.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:03.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:03.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:03.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:03.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:03.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2026-02-20T16:32:03.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:32:03.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 15 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 20 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:32:03.891 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:32:03.892 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-20T16:32:03.892 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:32:03.892 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-20T16:32:03.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-20T16:32:03.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:32:03.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:32:03.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:03.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2026-02-20T16:32:04.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:04.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2026-02-20T16:32:04.535 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2026-02-20T16:32:04.535 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:104778fc:::obj2:head# 2026-02-20T16:32:04.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:04.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:04.635 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:04.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:04.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:32:04.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:04.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:32:04.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:32:04.637 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:32:04.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:04.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:32:04.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:32:04.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:32:04.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:32:04.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:32:04.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:04.679+0000 7f8a31774a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:04.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:04.683+0000 7f8a31774a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:04.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:04.683+0000 7f8a31774a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:04.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:32:04.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:04.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:05.048 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:05.079+0000 7f8a31774a40 -1 Falling back to public interface 2026-02-20T16:32:05.680 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:05.711+0000 7f8a31774a40 -1 osd.1 42 log_to_monitors true 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:06.106 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [34,42) [v2:127.0.0.1:6810/3731123666,v1:127.0.0.1:6811/3731123666] [v2:127.0.0.1:6812/3731123666,v1:127.0.0.1:6813/3731123666] exists,up cd980eee-c89c-4efc-8f38-2ae86bc3479d 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:06.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:06.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:06.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:06.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:06.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:06.372 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:06.372 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:06.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:06.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:06.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:06.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691844 2026-02-20T16:32:06.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691844 2026-02-20T16:32:06.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844' 2026-02-20T16:32:06.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:06.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:06.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-20T16:32:06.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-20T16:32:06.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322' 2026-02-20T16:32:06.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:06.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:06.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574865 2026-02-20T16:32:06.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574865 2026-02-20T16:32:06.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322 2-55834574865' 2026-02-20T16:32:06.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:06.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691844 2026-02-20T16:32:06.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:06.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:06.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691844 2026-02-20T16:32:06.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:06.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691844 2026-02-20T16:32:06.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2026-02-20T16:32:06.596 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 171798691844 2026-02-20T16:32:06.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:06.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691844 2026-02-20T16:32:06.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:07.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:07.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691844 -lt 171798691844 2026-02-20T16:32:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:07.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528322 2026-02-20T16:32:07.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:07.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:07.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528322 2026-02-20T16:32:07.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:07.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-20T16:32:07.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2026-02-20T16:32:07.932 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 193273528322 2026-02-20T16:32:07.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:08.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-20T16:32:08.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:08.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574865 2026-02-20T16:32:08.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:08.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:08.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574865 2026-02-20T16:32:08.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:08.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574865 2026-02-20T16:32:08.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574865' 2026-02-20T16:32:08.097 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574865 2026-02-20T16:32:08.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:08.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574865 -lt 55834574865 2026-02-20T16:32:08.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:08.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:08.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:08.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:08.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:08.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:08.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:08.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:08.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:08.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:08.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:08.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:653: TEST_auto_repair_bluestore_failed: repair 1.0 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:32:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:32:08.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:32:08.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:32:08.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:08.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T16:32:09.120 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-20T16:32:09.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:09.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:32:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:32:09.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:32:09.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:32:09.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:51.638345+0000 '>' 2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:09.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:32:10.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:32:10.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:32:10.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:32:10.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:32:10.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:32:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:32:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:32:10.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:51.638345+0000 '>' 2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:10.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:32:11.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:32:11.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:32:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:32:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:32:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:32:11.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:32:11.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:32:11.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:31:51.638345+0000 '>' 2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:11.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:32:12.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:32:12.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:32:12.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:32:12.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:32:12.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:32:12.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:32:12.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:32:12.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:32:09.749509+0000 '>' 2026-02-20T16:31:51.638345+0000 2026-02-20T16:32:12.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:32:12.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_auto_repair_bluestore_failed: sleep 2 2026-02-20T16:32:14.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-20T16:32:14.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:14.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:14.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:14.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691847 2026-02-20T16:32:14.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691847 2026-02-20T16:32:14.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847' 2026-02-20T16:32:14.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:14.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:15.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-20T16:32:15.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-20T16:32:15.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528324' 2026-02-20T16:32:15.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:15.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:15.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574867 2026-02-20T16:32:15.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574867 2026-02-20T16:32:15.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528324 2-55834574867' 2026-02-20T16:32:15.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:15.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691847 2026-02-20T16:32:15.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:15.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:15.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691847 2026-02-20T16:32:15.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:15.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691847 2026-02-20T16:32:15.148 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 171798691847 2026-02-20T16:32:15.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691847' 2026-02-20T16:32:15.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:15.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691845 -lt 171798691847 2026-02-20T16:32:15.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:16.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691847 -lt 171798691847 2026-02-20T16:32:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:16.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528324 2026-02-20T16:32:16.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:16.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:16.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528324 2026-02-20T16:32:16.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:16.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-20T16:32:16.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528324' 2026-02-20T16:32:16.476 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 193273528324 2026-02-20T16:32:16.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:16.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528325 -lt 193273528324 2026-02-20T16:32:16.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:16.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574867 2026-02-20T16:32:16.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:16.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:16.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574867 2026-02-20T16:32:16.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:16.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574867 2026-02-20T16:32:16.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574867' 2026-02-20T16:32:16.649 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574867 2026-02-20T16:32:16.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:16.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574867 -lt 55834574867 2026-02-20T16:32:16.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-20T16:32:16.966 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:32:16.966 INFO:tasks.workunit.client.0.trial013.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2026-02-20T16:32:09.749567+0000 19'10 46:150 [1,0] 1 [1,0] 1 19'10 2026-02-20T16:32:09.749509+0000 19'10 2026-02-20T16:32:09.749509+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:32:09.749509+0000 9 0 2026-02-20T16:32:16.966 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:32:16.966 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:32:16.966 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:32:16.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-20T16:32:16.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2026-02-20T16:32:17.136 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:32:17.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 15 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 20 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:50.619+0000 7f87a76fd640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 19 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:31:51.635+0000 7f87a36f5640 10 osd.1 pg_epoch: 36 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:32:09.747+0000 7f8a0eef1640 10 osd.1 pg_epoch: 46 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=45/46 n=10 ec=15/15 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:32:09.747+0000 7f8a0eef1640 10 osd.1 pg_epoch: 46 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=45/46 n=10 ec=15/15 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:32:09.747+0000 7f8a0eef1640 15 osd.1 pg_epoch: 46 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=45/46 n=9 ec=15/15 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:32:09.747+0000 7f8a0eef1640 20 osd.1 pg_epoch: 46 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=45/46 n=9 ec=15/15 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish All may be fixed 2026-02-20T16:32:17.147 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:32:09.747+0000 7f8a0eef1640 19 osd.1 pg_epoch: 46 pg[1.0( v 19'10 (0'0,19'10] local-lis/les=45/46 n=9 ec=15/15 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=19'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:17.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:17.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:17.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:32:17.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:32:17.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:32:17.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:32:17.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:32:17.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:32:17.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:32:17.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:32:17.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:32:17.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:32:17.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:32:17.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:32:17.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:32:17.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:32:17.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:32:17.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:32:17.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:32:17.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:32:17.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:32:17.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:17.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:17.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:17.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:32:17.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:32:17.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:32:17.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:32:17.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:32:17.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:32:17.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:32:17.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:32:17.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:32:17.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:32:17.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:32:17.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:32:17.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:32:17.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:32:17.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:32:17.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:32:17.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:32:17.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:32:17.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:32:17.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:32:17.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:32:17.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:32:17.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:32:17.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:32:17.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:17.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:32:17.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:32:17.356 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:32:17.357 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:32:17.358 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:32:17.358 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.358 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.358 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:32:17.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:32:17.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:32:17.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:32:17.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:32:17.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:32:17.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:32:17.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:32:17.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:32:17.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:32:17.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:32:17.415 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:32:17.416 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.416 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.416 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:32:17.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:32:17.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:32:17.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:32:17.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:32:17.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:17.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:32:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:32:17.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:17.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2026-02-20T16:32:17.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-20T16:32:17.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:17.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:17.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:17.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:32:17.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:32:17.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:17.598 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:17.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8ae66d4b-0184-48e6-9e8c-4b42c11be49d' 2026-02-20T16:32:17.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:32:17.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRjJhpngoEJhAAkED+h8YiP7DK4w0yLRkbfA== 2026-02-20T16:32:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRjJhpngoEJhAAkED+h8YiP7DK4w0yLRkbfA=="}' 2026-02-20T16:32:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ae66d4b-0184-48e6-9e8c-4b42c11be49d -i td/osd-scrub-repair/0/new.json 2026-02-20T16:32:17.691 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:32:17.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:32:17.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCRjJhpngoEJhAAkED+h8YiP7DK4w0yLRkbfA== --osd-uuid 8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:17.717 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:17.747+0000 7f488d084a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:17.718 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:17.751+0000 7f488d084a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:17.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:17.751+0000 7f488d084a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:17.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:17.751+0000 7f488d084a40 -1 bdev(0x562f9fdeec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:32:17.720 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:17.751+0000 7f488d084a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:32:18.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:32:18.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:32:18.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:32:18.511 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:32:18.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:32:18.606 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:32:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:32:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:32:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:32:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:32:18.621 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:18.651+0000 7f49d3124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:18.623 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:18.655+0000 7f49d3124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:18.624 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:18.655+0000 7f49d3124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:18.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:19.060 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:19.091+0000 7f49d3124a40 -1 Falling back to public interface 2026-02-20T16:32:19.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:19.791 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:19.823+0000 7f49d3124a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:32:19.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:20.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:20.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:20.883 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:32:20.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:32:20.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:20.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:21.046 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/245573642,v1:127.0.0.1:6801/245573642] [v2:127.0.0.1:6802/245573642,v1:127.0.0.1:6803/245573642] exists,up 8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:21.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:32:21.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:32:21.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:21.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:32:21.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:32:21.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:21.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c38fefe6-9ed6-4886-82d6-bfdc07305f65' 2026-02-20T16:32:21.054 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:21.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:32:21.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCVjJhpBirQBRAA9p5djRFp8o+AqHXisvbl2Q== 2026-02-20T16:32:21.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCVjJhpBirQBRAA9p5djRFp8o+AqHXisvbl2Q=="}' 2026-02-20T16:32:21.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c38fefe6-9ed6-4886-82d6-bfdc07305f65 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:32:21.226 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:32:21.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:32:21.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCVjJhpBirQBRAA9p5djRFp8o+AqHXisvbl2Q== --osd-uuid c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:21.253 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:21.283+0000 7f291214fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:21.255 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:21.287+0000 7f291214fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:21.255 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:21.287+0000 7f291214fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:21.256 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:21.287+0000 7f291214fa40 -1 bdev(0x55ae69e08c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:32:21.256 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:21.287+0000 7f291214fa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:32:21.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:32:21.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:32:21.877 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:32:21.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:32:21.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:32:22.082 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:32:22.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:32:22.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:22.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:32:22.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:32:22.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:32:22.099 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:22.131+0000 7f4a5de02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:22.101 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:22.131+0000 7f4a5de02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:22.101 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:22.131+0000 7f4a5de02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:22.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:22.515 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:22.547+0000 7f4a5de02a40 -1 Falling back to public interface 2026-02-20T16:32:23.125 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:23.155+0000 7f4a5de02a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:23.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:23.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:24.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:24.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:24.562 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:32:24.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:32:24.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:24.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:24.721 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2542785068,v1:127.0.0.1:6811/2542785068] [v2:127.0.0.1:6812/2542785068,v1:127.0.0.1:6813/2542785068] exists,up c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:24.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:32:24.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:24.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:24.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:24.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:24.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:32:24.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:24.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:32:24.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:32:24.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c5b28674-e7fe-449b-b411-c5e1d6985eb3 2026-02-20T16:32:24.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c5b28674-e7fe-449b-b411-c5e1d6985eb3' 2026-02-20T16:32:24.731 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 c5b28674-e7fe-449b-b411-c5e1d6985eb3 2026-02-20T16:32:24.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:32:24.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYjJhpGn8cLhAAKZWcr3R/VWQc6CYVKnZmMQ== 2026-02-20T16:32:24.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYjJhpGn8cLhAAKZWcr3R/VWQc6CYVKnZmMQ=="}' 2026-02-20T16:32:24.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c5b28674-e7fe-449b-b411-c5e1d6985eb3 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:32:24.907 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:32:24.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:32:24.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCYjJhpGn8cLhAAKZWcr3R/VWQc6CYVKnZmMQ== --osd-uuid c5b28674-e7fe-449b-b411-c5e1d6985eb3 2026-02-20T16:32:24.935 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:24.967+0000 7fe96dba1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:24.937 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:24.967+0000 7fe96dba1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:24.938 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:24.971+0000 7fe96dba1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:24.938 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:24.971+0000 7fe96dba1a40 -1 bdev(0x55f7db436c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:32:24.938 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:24.971+0000 7fe96dba1a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:32:25.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:32:25.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:32:25.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:32:25.581 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:32:25.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:32:25.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:32:25.792 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:32:25.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:25.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:32:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:32:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:32:25.818 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:25.847+0000 7f882e635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:25.819 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:25.851+0000 7f882e635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:25.820 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:25.851+0000 7f882e635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:25.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:32:26.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:26.243 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:26.275+0000 7f882e635a40 -1 Falling back to public interface 2026-02-20T16:32:26.864 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:26.895+0000 7f882e635a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:32:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:27.116 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:32:27.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:27.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:32:27.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:28.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:32:28.437 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1298367755,v1:127.0.0.1:6819/1298367755] [v2:127.0.0.1:6820/1298367755,v1:127.0.0.1:6821/1298367755] exists,up c5b28674-e7fe-449b-b411-c5e1d6985eb3 2026-02-20T16:32:28.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:28.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:28.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:28.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2026-02-20T16:32:28.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-20T16:32:28.652 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:32:28.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:32:29.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2026-02-20T16:32:29.817 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:32:29.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-20T16:32:29.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:29.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:29.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:29.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:29.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:29.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:29.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:29.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:29.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:29.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:29.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:29.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:29.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:29.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:29.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:30.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:30.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:32:30.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:32:30.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:32:30.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:30.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:32:30.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:32:30.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-20T16:32:30.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:30.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:30.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-20T16:32:30.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-20T16:32:30.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-20T16:32:30.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:30.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:32:30.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:30.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:30.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:32:30.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:32:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:32:30.331 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:32:30.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:30.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:32:30.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:31.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:31.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:31.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:32:31.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:32.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:32:32.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:32.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:32:32.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:32.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:32:32.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:32.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:32.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:32:32.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:32.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:32:32.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:32:32.819 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:32:32.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:32.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T16:32:32.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:32.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-20T16:32:32.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:32.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:32.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-20T16:32:32.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:32.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-20T16:32:32.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-20T16:32:32.976 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542146 2026-02-20T16:32:32.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:33.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-20T16:32:33.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:33.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:33.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:33.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:33.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:33.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:32:33.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:33.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:33.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:33.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:32:33.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T16:32:33.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:32:33.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T16:32:33.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:32:33.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:32:33.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:32:33.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:33.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:32:33.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:33.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:33.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:32:34.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:32:34.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:32:34.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:32:34.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:32:34.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:32:34.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:34.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:34.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:34.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:34.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:32:34.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:34.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:34.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:34.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:32:34.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:32:34.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:32:34.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:32:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:32:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:32:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:32:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:32:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:32:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:32:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:35.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:35.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:32:35.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:35.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:35.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:35.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:32:35.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:32:35.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:32:35.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:32:36.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:32:36.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:32:36.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T16:32:36.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:32:36.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T16:32:36.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:32:36.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:36.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:36.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:36.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:36.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:36.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:36.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:37.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:37.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:37.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:37.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:37.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:681: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2026-02-20T16:32:37.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:682: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2026-02-20T16:32:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2026-02-20T16:32:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-20T16:32:37.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:32:37.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-20T16:32:37.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:32:37.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:32:37.457 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:32:37.457 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:32:37.457 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:32:37.457 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:37.457 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:32:37.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:32:37.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:37.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:32:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:32:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:32:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:37.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:37.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:37.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:32:38.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-20T16:32:38.277 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:38.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:38.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:32:38.360 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:32:38.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:38.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:32:38.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:32:38.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:32:38.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:32:38.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:32:38.380 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:38.410+0000 7febd54cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:38.381 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:38.410+0000 7febd54cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:38.382 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:38.414+0000 7febd54cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:38.521 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:32:38.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:38.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:38.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:38.824 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:38.854+0000 7febd54cfa40 -1 Falling back to public interface 2026-02-20T16:32:39.472 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:39.502+0000 7febd54cfa40 -1 osd.0 19 log_to_monitors true 2026-02-20T16:32:39.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:39.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:39.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:39.683 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:39.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:39.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:39.840 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 19 last_clean_interval [5,19) [v2:127.0.0.1:6800/1901710783,v1:127.0.0.1:6801/1901710783] [v2:127.0.0.1:6802/1901710783,v1:127.0.0.1:6803/1901710783] exists,up 8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:39.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:39.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:39.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:39.841 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:39.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:39.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:39.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:39.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:39.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:39.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:39.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:39.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:39.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:40.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:40.116 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:40.117 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:40.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:40.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:40.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:40.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:32:40.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:32:40.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:32:40.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:40.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:40.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-20T16:32:40.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-20T16:32:40.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672966' 2026-02-20T16:32:40.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:40.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:40.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542149 2026-02-20T16:32:40.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542149 2026-02-20T16:32:40.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672966 2-60129542149' 2026-02-20T16:32:40.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:40.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:32:40.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:40.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:40.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:32:40.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:40.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:32:40.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:32:40.341 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247810 2026-02-20T16:32:40.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:40.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:32:40.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:41.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:41.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:41.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:32:41.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:42.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:32:42.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:42.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-20T16:32:42.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:42.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-20T16:32:42.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:42.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:42.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-20T16:32:42.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:42.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-20T16:32:42.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-20T16:32:42.832 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672966 2026-02-20T16:32:42.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:42.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-20T16:32:42.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:42.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542149 2026-02-20T16:32:42.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:43.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:43.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542149 2026-02-20T16:32:43.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:43.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542149 2026-02-20T16:32:43.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2026-02-20T16:32:43.003 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542149 2026-02-20T16:32:43.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:43.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542149 2026-02-20T16:32:43.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:43.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:43.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:43.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:43.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:43.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:43.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:43.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:43.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:43.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:43.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:43.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:43.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:43.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:43.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:43.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:32:43.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-20T16:32:43.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:32:43.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:32:43.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:32:43.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:32:43.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:43.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:43.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:43.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:43.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:44.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:44.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:44.482 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:32:44.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:44.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:32:44.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:32:44.506 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:44.538+0000 7fb1df467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:44.508 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:44.538+0000 7fb1df467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:44.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:44.538+0000 7fb1df467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:44.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:44.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:44.639 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:32:44.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:44.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:44.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:44.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:44.910+0000 7fb1df467a40 -1 Falling back to public interface 2026-02-20T16:32:45.544 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:45.574+0000 7fb1df467a40 -1 osd.1 24 log_to_monitors true 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:45.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [10,24) [v2:127.0.0.1:6810/2742525854,v1:127.0.0.1:6811/2742525854] [v2:127.0.0.1:6812/2742525854,v1:127.0.0.1:6813/2742525854] exists,up c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:45.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:45.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:45.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:45.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:45.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:45.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:45.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:46.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:46.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:46.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:46.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:46.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:46.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:46.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:46.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247812 2026-02-20T16:32:46.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247812 2026-02-20T16:32:46.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812' 2026-02-20T16:32:46.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:46.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:46.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-20T16:32:46.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-20T16:32:46.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-120259084290' 2026-02-20T16:32:46.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:46.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:46.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-20T16:32:46.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-20T16:32:46.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-120259084290 2-60129542151' 2026-02-20T16:32:46.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:46.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247812 2026-02-20T16:32:46.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:46.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:46.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247812 2026-02-20T16:32:46.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:46.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247812 2026-02-20T16:32:46.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2026-02-20T16:32:46.452 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247812 2026-02-20T16:32:46.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:46.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247812 2026-02-20T16:32:46.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:47.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:47.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:47.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247812 2026-02-20T16:32:47.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:32:48.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:48.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247812 2026-02-20T16:32:48.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:48.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-20T16:32:48.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:48.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:48.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-20T16:32:48.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:48.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-20T16:32:48.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-20T16:32:48.950 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 120259084290 2026-02-20T16:32:48.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:49.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-20T16:32:49.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:49.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-20T16:32:49.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:49.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:49.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-20T16:32:49.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:49.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-20T16:32:49.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-20T16:32:49.110 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542151 2026-02-20T16:32:49.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:49.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-20T16:32:49.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:49.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:49.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:49.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:49.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:49.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:49.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:49.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:49.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:49.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:49.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:49.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:32:49.825 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:32:49.826 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:49.826 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:32:49.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:32:49.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:49.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:32:50.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:50.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:50.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:50.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:32:50.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-20T16:32:50.657 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:104778fc:::obj2:head# 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:32:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:50.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:50.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:32:50.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:32:50.757 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:32:50.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:50.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:32:50.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:32:50.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:32:50.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:32:50.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:32:50.778 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:50.810+0000 7ff945936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:50.779 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:50.810+0000 7ff945936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:50.780 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:50.810+0000 7ff945936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:32:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:32:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:50.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:50.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:50.921 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:32:50.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:50.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:51.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:51.231 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:51.262+0000 7ff945936a40 -1 Falling back to public interface 2026-02-20T16:32:51.877 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:51.906+0000 7ff945936a40 -1 osd.0 30 log_to_monitors true 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:52.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 34 up_thru 25 down_at 31 last_clean_interval [23,30) [v2:127.0.0.1:6800/2896147277,v1:127.0.0.1:6801/2896147277] [v2:127.0.0.1:6802/2896147277,v1:127.0.0.1:6803/2896147277] exists,up 8ae66d4b-0184-48e6-9e8c-4b42c11be49d 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:52.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:52.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:52.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:52.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:52.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-20T16:32:52.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-20T16:32:52.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-20T16:32:52.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:52.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:52.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084292 2026-02-20T16:32:52.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084292 2026-02-20T16:32:52.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-120259084292' 2026-02-20T16:32:52.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:52.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:52.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-20T16:32:52.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-20T16:32:52.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-120259084292 2-60129542153' 2026-02-20T16:32:52.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:52.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-20T16:32:52.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:52.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:52.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-20T16:32:52.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:52.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-20T16:32:52.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-20T16:32:52.727 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 146028888066 2026-02-20T16:32:52.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:52.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 146028888066 2026-02-20T16:32:52.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:53.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:53.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:54.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-20T16:32:54.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:54.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084292 2026-02-20T16:32:54.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:54.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:54.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084292 2026-02-20T16:32:54.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084292 2026-02-20T16:32:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084292' 2026-02-20T16:32:54.063 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 120259084292 2026-02-20T16:32:54.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:54.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084292 2026-02-20T16:32:54.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:54.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-20T16:32:54.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:54.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:54.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-20T16:32:54.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:54.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-20T16:32:54.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-20T16:32:54.238 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542153 2026-02-20T16:32:54.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:54.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-20T16:32:54.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:54.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:54.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:54.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:54.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:54.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:54.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:54.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:54.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:54.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:54.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:54.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-20T16:32:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:32:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:32:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:32:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:32:55.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:32:55.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:32:55.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:55.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-20T16:32:55.607 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2026-02-20T16:32:55.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:55.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:32:55.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:32:55.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:32:55.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:32:55.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:32:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:32:55.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:32:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:32:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:32:55.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:32:55.733 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:55.762+0000 7fbe4d9e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:55.735 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:55.766+0000 7fbe4d9e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:55.735 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:55.766+0000 7fbe4d9e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:55.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:56.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:32:56.127 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:56.158+0000 7fbe4d9e7a40 -1 Falling back to public interface 2026-02-20T16:32:56.727 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:32:56.758+0000 7fbe4d9e7a40 -1 osd.1 35 log_to_monitors true 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:32:57.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [28,35) [v2:127.0.0.1:6810/1733255078,v1:127.0.0.1:6811/1733255078] [v2:127.0.0.1:6812/1733255078,v1:127.0.0.1:6813/1733255078] exists,up c38fefe6-9ed6-4886-82d6-bfdc07305f65 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:32:57.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:32:57.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:32:57.191 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:32:57.192 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:32:57.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:32:57.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:32:57.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:32:57.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:32:57.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:32:57.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:32:57.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:32:57.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:32:57.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:32:57.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:32:57.456 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:32:57.457 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:32:57.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:32:57.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:57.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:32:57.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888068 2026-02-20T16:32:57.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888068 2026-02-20T16:32:57.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068' 2026-02-20T16:32:57.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:57.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:32:57.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-20T16:32:57.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-20T16:32:57.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068 1-167503724546' 2026-02-20T16:32:57.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:32:57.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:32:57.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-20T16:32:57.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-20T16:32:57.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068 1-167503724546 2-60129542155' 2026-02-20T16:32:57.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:57.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888068 2026-02-20T16:32:57.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:57.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:32:57.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888068 2026-02-20T16:32:57.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:57.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888068 2026-02-20T16:32:57.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2026-02-20T16:32:57.680 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 146028888068 2026-02-20T16:32:57.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:57.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888068 2026-02-20T16:32:57.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:32:58.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:32:58.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:32:59.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888068 -lt 146028888068 2026-02-20T16:32:59.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:59.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724546 2026-02-20T16:32:59.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:59.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:32:59.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724546 2026-02-20T16:32:59.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:59.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-20T16:32:59.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2026-02-20T16:32:59.014 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 167503724546 2026-02-20T16:32:59.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:32:59.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-20T16:32:59.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:32:59.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-20T16:32:59.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:32:59.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:32:59.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-20T16:32:59.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:32:59.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-20T16:32:59.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-20T16:32:59.174 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542155 2026-02-20T16:32:59.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:32:59.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-20T16:32:59.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:32:59.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:59.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:32:59.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:32:59.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:32:59.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:32:59.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:32:59.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:32:59.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:32:59.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:32:59.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:32:59.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-20T16:32:59.962 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T16:32:59.963 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T16:32:59.963 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:32:59.972 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T16:32:59.972 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T16:32:59.972 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:32:59.979 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T16:32:59.979 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-20T16:32:59.979 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:32:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2026-02-20T16:32:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:32:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-20T16:32:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-20T16:32:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:33:00.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2026-02-20T16:33:00.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2026-02-20T16:33:00.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:33:00.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:33:00.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:33:00.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:33:00.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2026-02-20T16:33:00.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2026-02-20T16:33:00.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:00.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:00.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:00.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:00.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2026-02-20T16:32:28.686217+0000 2026-02-20T16:33:00.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2026-02-20T16:33:00.534 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:33:00.535 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:33:00.535 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:33:00.535 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T11:44:50.569641+0000" 2026-02-20T16:33:00.535 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:33:00.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2026-02-20T16:32:28.686217+0000 2026-02-20T16:33:00.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:33:00.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:32:28.686217+0000 2026-02-20T16:33:00.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:00.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:00.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:32:28.686217+0000 '>' 2026-02-20T16:32:28.686217+0000 2026-02-20T16:33:00.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:33:01.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:33:01.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:33:01.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:33:01.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:01.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:01.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:01.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:33:00.784614+0000 '>' 2026-02-20T16:32:28.686217+0000 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:705: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:33:01.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:33:01.869 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:33:01.869 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:33:01.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:33:01.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:33:01.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:33:01.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:33:01.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:33:01.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:33:01.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:33:01.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:01.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:02.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:02.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888070 2026-02-20T16:33:02.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888070 2026-02-20T16:33:02.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888070' 2026-02-20T16:33:02.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:02.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:02.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724548 2026-02-20T16:33:02.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724548 2026-02-20T16:33:02.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888070 1-167503724548' 2026-02-20T16:33:02.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:02.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:02.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-20T16:33:02.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-20T16:33:02.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888070 1-167503724548 2-60129542157' 2026-02-20T16:33:02.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:02.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888070 2026-02-20T16:33:02.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:02.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:02.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888070 2026-02-20T16:33:02.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:02.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888070 2026-02-20T16:33:02.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888070' 2026-02-20T16:33:02.361 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 146028888070 2026-02-20T16:33:02.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:02.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888068 -lt 146028888070 2026-02-20T16:33:02.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:03.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:33:03.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:03.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888068 -lt 146028888070 2026-02-20T16:33:03.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:04.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:33:04.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:04.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888070 -lt 146028888070 2026-02-20T16:33:04.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:04.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724548 2026-02-20T16:33:04.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:04.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:04.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724548 2026-02-20T16:33:04.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:04.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724548 2026-02-20T16:33:04.860 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 167503724548 2026-02-20T16:33:04.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724548' 2026-02-20T16:33:04.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:05.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724548 -lt 167503724548 2026-02-20T16:33:05.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:05.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-20T16:33:05.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:05.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:05.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-20T16:33:05.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:05.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-20T16:33:05.025 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542157 2026-02-20T16:33:05.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-20T16:33:05.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:05.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542158 -lt 60129542157 2026-02-20T16:33:05.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:33:05.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:05.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:05.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:05.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:05.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:33:05.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:05.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:05.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:05.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:05.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:05.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888071 2026-02-20T16:33:05.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888071 2026-02-20T16:33:05.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071' 2026-02-20T16:33:05.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:05.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:06.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724549 2026-02-20T16:33:06.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724549 2026-02-20T16:33:06.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071 1-167503724549' 2026-02-20T16:33:06.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:06.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542159 2026-02-20T16:33:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542159 2026-02-20T16:33:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071 1-167503724549 2-60129542159' 2026-02-20T16:33:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:06.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888071 2026-02-20T16:33:06.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888071 2026-02-20T16:33:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:06.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888071 2026-02-20T16:33:06.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888071' 2026-02-20T16:33:06.110 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 146028888071 2026-02-20T16:33:06.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:06.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888070 -lt 146028888071 2026-02-20T16:33:06.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:07.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:33:07.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:07.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888070 -lt 146028888071 2026-02-20T16:33:07.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:08.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:33:08.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:08.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888072 -lt 146028888071 2026-02-20T16:33:08.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:08.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724549 2026-02-20T16:33:08.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:08.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:08.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724549 2026-02-20T16:33:08.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:08.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724549 2026-02-20T16:33:08.595 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 167503724549 2026-02-20T16:33:08.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2026-02-20T16:33:08.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:08.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724550 -lt 167503724549 2026-02-20T16:33:08.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:08.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542159 2026-02-20T16:33:08.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:08.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:08.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542159 2026-02-20T16:33:08.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:08.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542159 2026-02-20T16:33:08.772 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542159 2026-02-20T16:33:08.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-20T16:33:08.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:08.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542159 -lt 60129542159 2026-02-20T16:33:08.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2026-02-20T16:33:08.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-20T16:33:09.093 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:33:09.093 INFO:tasks.workunit.client.0.trial013.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-20T16:33:00.784663+0000 18'10 40:108 [1,0] 1 [1,0] 1 18'10 2026-02-20T16:33:00.784614+0000 18'10 2026-02-20T16:33:00.784614+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:33:00.784614+0000 10 0 2026-02-20T16:33:09.093 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:33:09.094 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:33:09.094 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:09.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-20T16:33:09.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2026-02-20T16:33:09.252 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:09.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:09.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:09.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:09.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:09.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:09.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:33:09.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:33:09.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:33:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:33:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:33:09.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:33:09.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:09.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:33:09.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:33:09.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:09.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:33:09.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:33:09.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:33:09.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:33:09.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:33:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:33:09.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:09.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:09.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:09.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:09.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:09.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:09.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:33:09.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:33:09.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:33:09.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:33:09.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:33:09.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:33:09.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:09.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:33:09.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:33:09.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:09.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:33:09.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:33:09.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:33:09.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:33:09.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:33:09.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:33:09.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:33:09.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:33:09.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:33:09.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:33:09.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:33:09.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:33:09.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:33:09.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:33:09.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:542: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:546: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:33:09.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:33:09.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:33:09.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:09.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:09.435 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:09.435 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.435 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:09.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:33:09.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:33:09.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:33:09.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:33:09.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:33:09.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:33:09.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:33:09.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:33:09.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:33:09.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:33:09.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:33:09.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:33:09.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:33:09.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:33:09.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:33:09.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:33:09.522 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:33:09.522 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:33:09.523 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.523 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.523 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:33:09.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:33:09.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:33:09.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2026-02-20T16:33:09.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:33:09.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:33:09.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:33:09.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:33:09.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:09.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:33:09.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:33:09.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:551: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:09.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: seq 0 2 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:09.695 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:09.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:33:09.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:33:09.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:09.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dfa2f2ba-5fce-4204-82aa-2d93abdda42f' 2026-02-20T16:33:09.698 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:09.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:33:09.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFjJhpjDA4LBAAPGiyRwQj2ILLtbLeBQ/dqw== 2026-02-20T16:33:09.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFjJhpjDA4LBAAPGiyRwQj2ILLtbLeBQ/dqw=="}' 2026-02-20T16:33:09.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dfa2f2ba-5fce-4204-82aa-2d93abdda42f -i td/osd-scrub-repair/0/new.json 2026-02-20T16:33:09.797 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:09.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:33:09.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDFjJhpjDA4LBAAPGiyRwQj2ILLtbLeBQ/dqw== --osd-uuid dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:09.824 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:09.854+0000 7f76eceb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:09.826 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:09.858+0000 7f76eceb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:09.827 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:09.858+0000 7f76eceb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:09.827 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:09.858+0000 7f76eceb5a40 -1 bdev(0x559b71a2ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:33:09.828 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:09.858+0000 7f76eceb5a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:33:10.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:33:10.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:33:10.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:33:10.614 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:33:10.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:33:10.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:33:10.710 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:33:10.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:33:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:33:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:33:10.725 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:10.754+0000 7fc6a4638a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:10.726 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:10.758+0000 7fc6a4638a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:10.728 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:10.758+0000 7fc6a4638a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:10.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:33:10.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:10.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:33:10.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:10.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:10.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:10.802 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:10.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:10.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:10.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:10.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:11.139 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:11.170+0000 7fc6a4638a40 -1 Falling back to public interface 2026-02-20T16:33:11.835 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:11.866+0000 7fc6a4638a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:33:11.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:11.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:11.896 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:33:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:33:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:11.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:12.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:12.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:12.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:33:12.985 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:33:12.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:12.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:13.140 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2311351503,v1:127.0.0.1:6801/2311351503] [v2:127.0.0.1:6802/2311351503,v1:127.0.0.1:6803/2311351503] exists,up dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:13.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:33:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:13.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:13.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:33:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:33:13.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:33:13.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f25ce74f-7f4f-4594-aca8-24d44fcbea22 2026-02-20T16:33:13.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f25ce74f-7f4f-4594-aca8-24d44fcbea22' 2026-02-20T16:33:13.148 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 f25ce74f-7f4f-4594-aca8-24d44fcbea22 2026-02-20T16:33:13.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:33:13.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJjJhpeyBrCxAAluac2r33ecupxXudEQJOhQ== 2026-02-20T16:33:13.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJjJhpeyBrCxAAluac2r33ecupxXudEQJOhQ=="}' 2026-02-20T16:33:13.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f25ce74f-7f4f-4594-aca8-24d44fcbea22 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:33:13.317 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:33:13.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:33:13.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDJjJhpeyBrCxAAluac2r33ecupxXudEQJOhQ== --osd-uuid f25ce74f-7f4f-4594-aca8-24d44fcbea22 2026-02-20T16:33:13.344 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:13.374+0000 7fc0a5960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:13.346 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:13.378+0000 7fc0a5960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:13.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:13.378+0000 7fc0a5960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:13.348 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:13.378+0000 7fc0a5960a40 -1 bdev(0x55a80d892c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:33:13.348 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:13.378+0000 7fc0a5960a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:33:14.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:33:14.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:33:14.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:33:14.007 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:33:14.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:33:14.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:33:14.205 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:33:14.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:14.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:33:14.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:33:14.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:33:14.231 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:14.262+0000 7f68626ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:14.233 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:14.262+0000 7f68626ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:14.234 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:14.266+0000 7f68626ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:14.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:33:14.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:14.368 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:14.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:14.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:33:14.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:14.627 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:14.658+0000 7f68626ffa40 -1 Falling back to public interface 2026-02-20T16:33:15.263 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:15.294+0000 7f68626ffa40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:33:15.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:15.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:15.530 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:33:15.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:33:15.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:15.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:33:15.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:16.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:16.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:16.694 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:33:16.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:33:16.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:16.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:33:16.847 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/655576506,v1:127.0.0.1:6811/655576506] [v2:127.0.0.1:6812/655576506,v1:127.0.0.1:6813/655576506] exists,up f25ce74f-7f4f-4594-aca8-24d44fcbea22 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:33:16.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:16.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:16.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:16.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:33:16.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:16.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:33:16.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:33:16.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a6445aa-80de-4443-870f-75a410e098e6 2026-02-20T16:33:16.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3a6445aa-80de-4443-870f-75a410e098e6' 2026-02-20T16:33:16.855 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 3a6445aa-80de-4443-870f-75a410e098e6 2026-02-20T16:33:16.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:33:16.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMjJhp6kOWNRAAp09GKysAUh3GAySft2L5aw== 2026-02-20T16:33:16.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMjJhp6kOWNRAAp09GKysAUh3GAySft2L5aw=="}' 2026-02-20T16:33:16.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a6445aa-80de-4443-870f-75a410e098e6 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:33:17.022 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:33:17.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:33:17.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDMjJhp6kOWNRAAp09GKysAUh3GAySft2L5aw== --osd-uuid 3a6445aa-80de-4443-870f-75a410e098e6 2026-02-20T16:33:17.048 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.078+0000 7f6c9e201a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:17.050 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.082+0000 7f6c9e201a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:17.051 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.082+0000 7f6c9e201a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:17.052 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.082+0000 7f6c9e201a40 -1 bdev(0x55eec93a0c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:33:17.052 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.082+0000 7f6c9e201a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:33:17.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:33:17.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:33:17.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:33:17.702 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:33:17.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:33:17.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:33:17.894 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:33:17.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:17.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:33:17.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:33:17.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:33:17.911 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.942+0000 7efd7df17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:17.912 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.942+0000 7efd7df17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:17.913 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:17.942+0000 7efd7df17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:18.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:33:18.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:18.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:33:18.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:18.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:18.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:18.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:18.059 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:18.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:18.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:33:18.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:18.291 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:18.322+0000 7efd7df17a40 -1 Falling back to public interface 2026-02-20T16:33:18.927 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:18.958+0000 7efd7df17a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:33:19.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:19.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:19.216 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:33:19.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:33:19.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:19.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:33:19.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:20.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:20.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:20.375 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:33:20.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:33:20.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:20.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:33:20.531 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4178955674,v1:127.0.0.1:6819/4178955674] [v2:127.0.0.1:6820/4178955674,v1:127.0.0.1:6821/4178955674] exists,up 3a6445aa-80de-4443-870f-75a410e098e6 2026-02-20T16:33:20.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:33:20.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:33:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:33:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:556: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2026-02-20T16:33:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-20T16:33:20.735 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:33:20.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:33:21.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2026-02-20T16:33:21.947 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:33:21.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-20T16:33:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:33:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:33:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:33:21.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:33:21.959 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:33:21.959 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:33:21.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:33:21.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:33:21.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:22.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:22.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:22.238 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:22.238 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:22.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:22.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:22.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:22.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:33:22.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:33:22.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:33:22.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:22.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:22.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:33:22.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:33:22.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-20T16:33:22.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:22.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:22.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-20T16:33:22.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-20T16:33:22.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-20T16:33:22.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:22.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:33:22.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:22.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:22.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:33:22.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:22.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:33:22.458 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:33:22.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:33:22.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:22.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:33:22.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:23.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:33:23.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:23.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:33:23.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:24.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:33:24.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:24.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:33:24.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:24.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:33:24.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:24.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:24.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:33:24.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:24.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:33:24.944 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:33:24.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:33:24.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:25.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T16:33:25.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:25.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-20T16:33:25.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:25.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:25.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-20T16:33:25.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:25.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-20T16:33:25.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-20T16:33:25.116 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542146 2026-02-20T16:33:25.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:25.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-20T16:33:25.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:33:25.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:25.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:25.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:25.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:25.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:25.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:25.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:25.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T16:33:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:33:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T16:33:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:33:25.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:25.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:25.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:25.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:25.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:25.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:25.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:25.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:26.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:26.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:26.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:26.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:26.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:26.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:26.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:26.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:26.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:26.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:26.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:26.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:26.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:26.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:26.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:26.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:33:26.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:26.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:26.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:26.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:26.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:26.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:27.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:27.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:27.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:27.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:27.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:33:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:33:27.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:27.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:27.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:27.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:27.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:27.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:27.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:27.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:27.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:28.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:28.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:28.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T16:33:28.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:28.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T16:33:28.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:28.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:28.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:28.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:29.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:33:29.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:29.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:29.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:562: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2026-02-20T16:33:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:563: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:33:29.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-20T16:33:29.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:33:29.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:33:29.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:33:29.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:33:29.342 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:33:29.342 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:29.342 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:33:29.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:33:29.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:33:29.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:33:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:33:29.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:29.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:29.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:29.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:33:29.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:33:29.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:33:29.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:33:29.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:33:29.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:29.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T16:33:30.177 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:33:30.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:33:30.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:30.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:30.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:33:30.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:33:30.276 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:33:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:33:30.294 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:30.326+0000 7f36c4960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:30.295 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:30.326+0000 7f36c4960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:30.296 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:30.326+0000 7f36c4960a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:30.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:30.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:30.721+0000 7f36c4960a40 -1 Falling back to public interface 2026-02-20T16:33:31.291 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:31.321+0000 7f36c4960a40 -1 osd.0 21 log_to_monitors true 2026-02-20T16:33:31.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:31.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:31.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:33:31.600 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:31.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:31.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:31.757 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6800/3072026279,v1:127.0.0.1:6801/3072026279] [v2:127.0.0.1:6802/3072026279,v1:127.0.0.1:6803/3072026279] exists,up dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:33:31.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:33:31.759 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:33:31.759 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:33:31.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:33:31.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:33:31.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:31.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:32.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:32.027 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:32.027 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:32.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:32.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:32.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:32.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-20T16:33:32.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-20T16:33:32.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-20T16:33:32.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:32.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:32.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-20T16:33:32.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-20T16:33:32.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2026-02-20T16:33:32.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:32.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:32.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542149 2026-02-20T16:33:32.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542149 2026-02-20T16:33:32.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-60129542149' 2026-02-20T16:33:32.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:32.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-20T16:33:32.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:32.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:32.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-20T16:33:32.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-20T16:33:32.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-20T16:33:32.250 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 103079215106 2026-02-20T16:33:32.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-20T16:33:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:33:33.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:33.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-20T16:33:33.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:34.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:33:34.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:34.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-20T16:33:34.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:34.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-20T16:33:34.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:34.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:34.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-20T16:33:34.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:34.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-20T16:33:34.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-20T16:33:34.750 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672966 2026-02-20T16:33:34.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:34.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-20T16:33:34.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:34.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542149 2026-02-20T16:33:34.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:34.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:34.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542149 2026-02-20T16:33:34.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:34.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542149 2026-02-20T16:33:34.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2026-02-20T16:33:34.919 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542149 2026-02-20T16:33:34.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:35.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542149 2026-02-20T16:33:35.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:33:35.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:35.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:35.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:33:35.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:35.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:35.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:35.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:35.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:35.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:35.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:35.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:33:35.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:35.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:35.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:35.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:33:35.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:33:35.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:33:35.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2026-02-20T16:33:35.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:33:35.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:33:35.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:35.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:33:35.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2026-02-20T16:33:35.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2026-02-20T16:33:35.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:33:35.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:33:35.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:35.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:33:35.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: local primary=1 2026-02-20T16:33:35.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2026-02-20T16:33:35.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:35.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:35.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:35.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2026-02-20T16:33:36.175 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:33:36.175 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:33:36.175 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:33:36.175 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:33:36.209799+0000" 2026-02-20T16:33:36.175 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:575: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:33:36.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:33:36.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:33:36.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:36.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:36.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:36.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:36.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:33:20.769372+0000 '>' 2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:36.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:33:37.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:33:37.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:33:37.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:33:37.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:37.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:37.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:37.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:37.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:33:20.769372+0000 '>' 2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:37.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:33:38.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:33:38.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:33:38.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:33:38.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:33:38.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:33:38.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:33:38.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:33:38.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:33:36.456335+0000 '>' 2026-02-20T16:33:20.769372+0000 2026-02-20T16:33:38.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:33:38.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:576: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-20T16:33:38.817 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:33:38.817 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+inconsistent 2026-02-20T16:33:36.456396+0000 20'1 25:54 [1,0] 1 [1,0] 1 20'1 2026-02-20T16:33:36.456335+0000 0'0 2026-02-20T16:33:20.769372+0000 0 0 queued for deep scrub 1 0 2026-02-20T16:33:38.817 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:33:38.818 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:33:38.818 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:38.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-20T16:33:40.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-20T16:33:40.975 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:33:40.975 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+inconsistent 2026-02-20T16:33:36.456396+0000 20'1 25:54 [1,0] 1 [1,0] 1 20'1 2026-02-20T16:33:36.456335+0000 0'0 2026-02-20T16:33:20.769372+0000 0 0 queued for deep scrub 1 0 2026-02-20T16:33:40.975 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:33:40.975 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:33:40.976 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:40.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-20T16:33:42.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-20T16:33:43.136 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:33:43.137 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-20T16:33:37.499324+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2026-02-20T16:33:37.497726+0000 20'1 2026-02-20T16:33:37.497726+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:33:37.497726+0000 1 0 2026-02-20T16:33:43.137 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:33:43.137 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:33:43.137 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:43.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_scrub: sleep 5 2026-02-20T16:33:48.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-20T16:33:48.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:33:48.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:33:48.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:33:48.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:33:48.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:33:48.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:33:48.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:33:48.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:33:48.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:33:48.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:33:48.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:33:48.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:33:48.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:33:48.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:48.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:48.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:48.416 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:48.416 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:48.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:48.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:48.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215110 2026-02-20T16:33:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215110 2026-02-20T16:33:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110' 2026-02-20T16:33:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:48.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-20T16:33:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-20T16:33:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110 1-42949672970' 2026-02-20T16:33:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:48.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:48.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-20T16:33:48.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-20T16:33:48.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110 1-42949672970 2-60129542153' 2026-02-20T16:33:48.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:48.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215110 2026-02-20T16:33:48.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:48.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:48.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215110 2026-02-20T16:33:48.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:48.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215110 2026-02-20T16:33:48.638 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 103079215110 2026-02-20T16:33:48.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2026-02-20T16:33:48.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215109 -lt 103079215110 2026-02-20T16:33:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:33:49.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:33:49.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:49.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215110 -lt 103079215110 2026-02-20T16:33:49.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:49.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-20T16:33:49.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:49.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:49.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-20T16:33:49.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:49.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-20T16:33:49.964 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672970 2026-02-20T16:33:49.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-20T16:33:49.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:50.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-20T16:33:50.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:50.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-20T16:33:50.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:50.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-20T16:33:50.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:50.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-20T16:33:50.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-20T16:33:50.128 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542153 2026-02-20T16:33:50.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-20T16:33:50.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:33:50.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:50.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:50.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:33:50.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:50.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:50.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:50.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:33:50.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:33:50.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:33:50.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-20T16:33:50.956 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:33:50.956 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-20T16:33:37.499324+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2026-02-20T16:33:37.497726+0000 20'1 2026-02-20T16:33:37.497726+0000 0 0 periodic scrub scheduled @ 2026-02-21T16:33:37.497726+0000 1 0 2026-02-20T16:33:50.957 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:33:50.957 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:33:50.957 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:33:50.968 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:33:50.969 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:50.969 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:33:51.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:33:51.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:33:51.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:33:51.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:33:51.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:33:51.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:51.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:51.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:51.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:51.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-20T16:33:51.684 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:33:51.685 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:33:51.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-20T16:33:51.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:33:51.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:33:51.772 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:33:51.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-20T16:33:51.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:33:51.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:33:51.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:33:51.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:33:51.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:33:51.791 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:51.821+0000 7f1c91fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:51.793 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:51.821+0000 7f1c91fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:51.794 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:51.825+0000 7f1c91fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:51.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:33:51.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:51.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:33:51.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:51.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:52.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:33:52.215 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:52.245+0000 7f1c91fdaa40 -1 Falling back to public interface 2026-02-20T16:33:52.901 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:52.929+0000 7f1c91fdaa40 -1 osd.0 25 log_to_monitors true 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:53.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:53.251 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6800/1725597372,v1:127.0.0.1:6801/1725597372] [v2:127.0.0.1:6802/1725597372,v1:127.0.0.1:6803/1725597372] exists,up dfa2f2ba-5fce-4204-82aa-2d93abdda42f 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:33:53.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:33:53.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:33:53.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:33:53.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:33:53.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:33:53.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:33:53.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:33:53.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:33:53.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:33:53.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:33:53.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:33:53.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:33:53.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:33:53.534 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:33:53.534 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:33:53.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:33:53.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:33:53.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-20T16:33:53.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-20T16:33:53.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-20T16:33:53.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:53.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:33:53.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672972 2026-02-20T16:33:53.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672972 2026-02-20T16:33:53.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672972' 2026-02-20T16:33:53.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:33:53.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:33:53.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-20T16:33:53.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-20T16:33:53.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672972 2-60129542155' 2026-02-20T16:33:53.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:53.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-20T16:33:53.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:53.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:33:53.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-20T16:33:53.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:53.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-20T16:33:53.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-20T16:33:53.756 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 124554051586 2026-02-20T16:33:53.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:33:53.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-20T16:33:53.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:53.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672972 2026-02-20T16:33:53.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:53.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:33:53.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672972 2026-02-20T16:33:53.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:53.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672972 2026-02-20T16:33:53.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-20T16:33:53.929 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672972 2026-02-20T16:33:53.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:33:54.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672972 -lt 42949672972 2026-02-20T16:33:54.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:33:54.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-20T16:33:54.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:33:54.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:33:54.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-20T16:33:54.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:33:54.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-20T16:33:54.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-20T16:33:54.094 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542155 2026-02-20T16:33:54.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:33:54.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-20T16:33:54.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:33:54.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:54.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:54.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:54.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:54.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:54.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:54.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:54.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:54.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:54.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T16:33:54.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:33:54.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T16:33:54.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:54.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:33:54.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:54.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:54.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:54.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:54.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:54.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:54.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:54.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:55.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:55.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:55.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:55.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:55.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:55.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:55.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:55.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:55.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:33:55.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:55.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:33:55.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:55.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:55.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:55.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:55.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:55.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:55.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:56.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:56.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:56.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:56.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:56.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:33:56.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:56.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:33:56.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:56.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:56.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T16:33:56.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:56.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:56.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T16:33:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T16:33:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:33:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:33:56.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:33:56.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:33:56.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:33:56.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:33:56.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:33:57.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:33:57.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:33:57.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T16:33:57.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:33:57.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T16:33:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:33:57.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:33:57.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:33:57.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:33:57.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:33:57.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:33:57.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:33:57.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:33:58.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:33:58.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:33:58.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:33:58.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:33:58.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:33:58.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:33:58.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:33:58.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:33:58.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:33:58.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:33:58.316 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:36.453+0000 7f683fe7c640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2026-02-20T16:33:58.317 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:36.453+0000 7f683fe7c640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish: will initiate a deep scrub to fix 1 errors 2026-02-20T16:33:58.317 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:36.453+0000 7f683fe7c640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-20T16:33:58.317 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:37.493+0000 7f683fe7c640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-20T16:33:58.317 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:37.493+0000 7f683fe7c640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:33:58.317 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:37.493+0000 7f683fe7c640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-20T16:33:58.318 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:37.493+0000 7f683fe7c640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-20T16:33:58.318 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:33:37.493+0000 7f683fe7c640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:33:58.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2026-02-20T16:33:58.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2026-02-20T16:33:58.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: COUNT=1 2026-02-20T16:33:58.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_scrub: test 1 = 1 2026-02-20T16:33:58.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:33:58.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:58.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:58.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:58.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:33:58.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:33:58.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:33:58.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:33:58.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:33:58.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:33:58.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:58.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:33:58.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:33:58.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:58.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:33:58.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:33:58.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:33:58.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:33:58.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:33:58.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:33:58.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:33:58.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:33:58.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:33:58.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:33:58.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:33:58.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:33:58.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:33:58.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:58.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:33:58.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:33:58.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:33:58.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:33:58.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:33:58.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:33:58.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:33:58.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:33:58.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:33:58.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:33:58.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:33:58.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:33:58.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:33:58.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:33:58.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:33:58.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:33:58.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:33:58.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2026-02-20T16:33:58.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:444: TEST_auto_repair_bluestore_tag: local poolname=testpool 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:33:58.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:58.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:33:58.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:33:58.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:33:58.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:33:58.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:33:58.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:33:58.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:33:58.594 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:33:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:33:58.653 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:33:58.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:33:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2026-02-20T16:33:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:33:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:33:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:33:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:33:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:33:58.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:33:58.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:58.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:58.806 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:58.806 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.806 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:58.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:33:58.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:33:58.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:33:58.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: seq 0 2 2026-02-20T16:33:58.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-20T16:33:58.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:33:58.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:33:58.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:33:58.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:33:58.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:33:58.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:33:58.835 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:33:58.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0deb9c59-ac96-4ee0-8b78-84dea313f230' 2026-02-20T16:33:58.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:33:58.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2jJhpSNw+NBAASvk/qldIKJnXcVA96KRxrw== 2026-02-20T16:33:58.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2jJhpSNw+NBAASvk/qldIKJnXcVA96KRxrw=="}' 2026-02-20T16:33:58.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0deb9c59-ac96-4ee0-8b78-84dea313f230 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:33:58.931 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:58.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:33:58.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQD2jJhpSNw+NBAASvk/qldIKJnXcVA96KRxrw== --osd-uuid 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:33:58.957 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:58.985+0000 7f5f67ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:58.958 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:58.989+0000 7f5f67ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:58.959 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:58.989+0000 7f5f67ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:58.959 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:58.989+0000 7f5f67ca6a40 -1 bdev(0x56456dbb8c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:33:58.959 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:58.989+0000 7f5f67ca6a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:33:59.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:33:59.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:33:59.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:33:59.675 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:33:59.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:33:59.780 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:33:59.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:33:59.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:33:59.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:33:59.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:33:59.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:33:59.807 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:59.837+0000 7f94af653a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:59.809 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:59.841+0000 7f94af653a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:59.811 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:33:59.841+0000 7f94af653a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:33:59.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:33:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:00.211 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:00.241+0000 7f94af653a40 -1 Falling back to public interface 2026-02-20T16:34:00.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:00.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:00.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:00.967 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:00.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:00.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:01.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:01.224 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:01.253+0000 7f94af653a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:34:02.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:02.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:02.063 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:02.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:34:02.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:02.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:02.223 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/305765087,v1:127.0.0.1:6801/305765087] [v2:127.0.0.1:6802/305765087,v1:127.0.0.1:6803/305765087] exists,up 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:34:02.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:02.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:02.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:02.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:02.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:34:02.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:34:02.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:34:02.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ca828c1a-f186-43dd-a59f-1ae1a0c0aadf 2026-02-20T16:34:02.231 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 ca828c1a-f186-43dd-a59f-1ae1a0c0aadf 2026-02-20T16:34:02.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ca828c1a-f186-43dd-a59f-1ae1a0c0aadf' 2026-02-20T16:34:02.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:34:02.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6jJhpuI5fEBAAc/p6VCkHfSYo9v41/gzanw== 2026-02-20T16:34:02.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6jJhpuI5fEBAAc/p6VCkHfSYo9v41/gzanw=="}' 2026-02-20T16:34:02.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ca828c1a-f186-43dd-a59f-1ae1a0c0aadf -i td/osd-scrub-repair/1/new.json 2026-02-20T16:34:02.406 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:02.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:34:02.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQD6jJhpuI5fEBAAc/p6VCkHfSYo9v41/gzanw== --osd-uuid ca828c1a-f186-43dd-a59f-1ae1a0c0aadf 2026-02-20T16:34:02.442 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:02.473+0000 7fa34b2cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:02.444 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:02.473+0000 7fa34b2cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:02.445 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:02.473+0000 7fa34b2cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:02.446 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:02.477+0000 7fa34b2cda40 -1 bdev(0x562559948c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:34:02.446 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:02.477+0000 7fa34b2cda40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:34:03.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:34:03.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:34:03.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:34:03.118 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:34:03.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:34:03.322 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:34:03.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:34:03.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:03.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:34:03.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:34:03.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:34:03.349 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:03.381+0000 7fb54ee9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:03.351 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:03.381+0000 7fb54ee9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:03.352 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:03.381+0000 7fb54ee9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:03.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:03.483 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:03.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:03.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:34:03.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:03.759 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:03.789+0000 7fb54ee9aa40 -1 Falling back to public interface 2026-02-20T16:34:04.391 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:04.421+0000 7fb54ee9aa40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:34:04.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:04.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:04.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:04.645 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:04.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:04.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:34:04.805 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4117510048,v1:127.0.0.1:6811/4117510048] [v2:127.0.0.1:6812/4117510048,v1:127.0.0.1:6813/4117510048] exists,up ca828c1a-f186-43dd-a59f-1ae1a0c0aadf 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:04.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:04.807 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:34:04.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:34:04.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=516323a4-ea1f-4cd4-9279-3c40904db6c9 2026-02-20T16:34:04.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 516323a4-ea1f-4cd4-9279-3c40904db6c9' 2026-02-20T16:34:04.810 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 516323a4-ea1f-4cd4-9279-3c40904db6c9 2026-02-20T16:34:04.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:34:04.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8jJhpytUdMxAAnqQlXpxjVIph9f3fS5h81Q== 2026-02-20T16:34:04.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8jJhpytUdMxAAnqQlXpxjVIph9f3fS5h81Q=="}' 2026-02-20T16:34:04.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 516323a4-ea1f-4cd4-9279-3c40904db6c9 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:34:04.984 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:04.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:34:04.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQD8jJhpytUdMxAAnqQlXpxjVIph9f3fS5h81Q== --osd-uuid 516323a4-ea1f-4cd4-9279-3c40904db6c9 2026-02-20T16:34:05.012 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.041+0000 7f802e620a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:05.014 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.045+0000 7f802e620a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:05.015 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.045+0000 7f802e620a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:05.015 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.045+0000 7f802e620a40 -1 bdev(0x557e98a16c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:34:05.015 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.045+0000 7f802e620a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:34:05.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:34:05.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:34:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:34:05.674 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:34:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:34:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:34:05.907 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.937+0000 7fdcf322ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:05.909 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.941+0000 7fdcf322ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:05.910 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:05.941+0000 7fdcf322ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:06.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:34:06.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:06.036 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:06.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:06.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:34:06.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:06.243 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:06.273+0000 7fdcf322ea40 -1 Falling back to public interface 2026-02-20T16:34:06.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:06.949+0000 7fdcf322ea40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:34:07.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:07.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:07.198 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:07.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:34:07.356 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/574096518,v1:127.0.0.1:6819/574096518] [v2:127.0.0.1:6820/574096518,v1:127.0.0.1:6821/574096518] exists,up 516323a4-ea1f-4cd4-9279-3c40904db6c9 2026-02-20T16:34:07.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:07.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:07.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:07.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2026-02-20T16:34:07.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-20T16:34:07.584 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:34:07.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:34:08.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2026-02-20T16:34:08.805 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:34:08.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_tag: wait_for_clean 2026-02-20T16:34:08.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:08.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:08.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:08.817 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:08.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:08.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:08.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:08.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:08.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:08.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:09.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:09.097 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:09.098 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:09.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:09.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:09.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:09.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:34:09.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:34:09.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:34:09.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:09.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:09.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-20T16:34:09.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-20T16:34:09.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-20T16:34:09.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:09.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:09.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-20T16:34:09.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-20T16:34:09.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666 2-55834574850' 2026-02-20T16:34:09.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:09.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:34:09.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:09.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:34:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:09.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:34:09.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:34:09.319 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:34:09.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:09.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:34:09.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:10.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:10.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:10.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:34:10.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:11.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:34:11.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:11.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:34:11.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:11.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-20T16:34:11.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:11.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:11.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-20T16:34:11.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:11.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-20T16:34:11.812 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705666 2026-02-20T16:34:11.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-20T16:34:11.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:11.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705666 2026-02-20T16:34:11.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-20T16:34:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:11.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:11.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-20T16:34:11.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:11.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-20T16:34:11.981 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574850 2026-02-20T16:34:11.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-20T16:34:11.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:12.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-20T16:34:12.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:34:12.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:12.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:12.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:34:12.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:34:12.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:34:12.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:34:12.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:34:12.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:34:12.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:34:12.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:34:12.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:34:12.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:34:12.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:12.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_tag: echo ABCDEF 2026-02-20T16:34:12.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:34:12.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-20T16:34:12.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:34:12.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:34:12.720 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:34:12.720 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:34:12.720 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:34:12.720 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:12.720 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:34:12.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:34:12.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:12.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:34:13.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:34:13.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:13.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T16:34:13.563 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:34:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:13.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:13.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:34:13.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:13.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:34:13.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:34:13.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:34:13.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:34:13.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:34:13.689 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:13.721+0000 7fac484caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:13.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:13.721+0000 7fac484caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:13.692 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:13.721+0000 7fac484caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:13.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:34:13.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:13.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:34:13.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:13.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:13.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:13.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:13.829 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:34:13.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:13.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:13.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:14.139 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:14.169+0000 7fac484caa40 -1 Falling back to public interface 2026-02-20T16:34:14.727 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:14.757+0000 7fac484caa40 -1 osd.0 19 log_to_monitors true 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:15.149 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/602359386,v1:127.0.0.1:6801/602359386] [v2:127.0.0.1:6802/602359386,v1:127.0.0.1:6803/602359386] exists,up 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:15.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:15.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:15.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:15.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:15.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:15.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:15.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:15.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:15.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:15.429 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:15.429 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:15.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:15.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:15.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:15.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:34:15.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:34:15.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:34:15.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:15.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:15.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:34:15.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:34:15.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705669' 2026-02-20T16:34:15.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:15.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-20T16:34:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-20T16:34:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705669 2-55834574852' 2026-02-20T16:34:15.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:15.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:34:15.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:15.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:15.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:34:15.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:15.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:34:15.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:34:15.652 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247810 2026-02-20T16:34:15.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:15.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:34:15.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:16.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:16.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:16.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:34:16.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:17.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:34:17.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:18.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-20T16:34:18.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:18.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:34:18.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:18.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:18.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:34:18.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:18.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:34:18.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:34:18.147 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705669 2026-02-20T16:34:18.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:18.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:34:18.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:18.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-20T16:34:18.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:18.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:18.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-20T16:34:18.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:18.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-20T16:34:18.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-20T16:34:18.309 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574852 2026-02-20T16:34:18.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:18.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-20T16:34:18.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:34:18.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:18.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:34:18.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:34:18.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:34:18.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:34:18.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:18.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:19.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:34:19.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:34:19.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:34:19.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2026-02-20T16:34:19.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:34:19.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:34:19.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:19.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:34:19.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: local pgid=1.0 2026-02-20T16:34:19.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2026-02-20T16:34:19.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:34:19.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:34:19.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:19.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:34:19.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: local primary=1 2026-02-20T16:34:19.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:475: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2026-02-20T16:34:19.362 INFO:tasks.workunit.client.0.trial013.stdout:Affected PG 1.0 w/ primary 1 2026-02-20T16:34:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2026-02-20T16:34:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:34:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:34:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:34:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2026-02-20T16:34:07.616498+0000 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: initiate_and_fetch_state: local the_osd=osd.1 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: initiate_and_fetch_state: local pgid=1.0 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:34:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:34:19.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: local last_scrub=2026-02-20T16:34:07.616498+0000 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:386: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-20T16:34:19.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:19.682 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:19.683 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:34:19.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2026-02-20T16:34:19.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:34:19.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:34:19.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set osd_scrub_auto_repair true 2026-02-20T16:34:19.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:34:19.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:389: initiate_and_fetch_state: flush_pg_stats 2026-02-20T16:34:19.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:19.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:19.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:20.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247812 2026-02-20T16:34:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247812 2026-02-20T16:34:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812' 2026-02-20T16:34:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:20.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:20.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-20T16:34:20.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-20T16:34:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-38654705670' 2026-02-20T16:34:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:20.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:20.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-20T16:34:20.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-20T16:34:20.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-38654705670 2-55834574854' 2026-02-20T16:34:20.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:20.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247812 2026-02-20T16:34:20.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:20.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:20.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247812 2026-02-20T16:34:20.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:20.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247812 2026-02-20T16:34:20.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2026-02-20T16:34:20.181 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 98784247812 2026-02-20T16:34:20.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:20.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247812 2026-02-20T16:34:20.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:21.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:21.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:21.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247812 2026-02-20T16:34:21.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:21.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-20T16:34:21.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:21.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:21.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-20T16:34:21.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:21.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-20T16:34:21.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-20T16:34:21.510 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705670 2026-02-20T16:34:21.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705671 -lt 38654705670 2026-02-20T16:34:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:21.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574854 2026-02-20T16:34:21.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:21.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:21.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574854 2026-02-20T16:34:21.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:21.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574854 2026-02-20T16:34:21.674 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574854 2026-02-20T16:34:21.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-20T16:34:21.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:21.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-20T16:34:21.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:390: initiate_and_fetch_state: date --rfc-3339=ns 2026-02-20T16:34:21.836 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20 16:34:21.871544295+00:00 2026-02-20T16:34:21.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: get_asok_path osd.1 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:34:21.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok schedule-deep-scrub 1.0 2026-02-20T16:34:21.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i=0 )) 2026-02-20T16:34:21.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:21.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:21.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:21.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:21.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2026-02-20T16:34:21.977 INFO:tasks.workunit.client.0.trial013.stdout:{"deep":true,"must":false,"stamp":"2026-02-09T11:46:11.930084+0000"}0 ) state now: "active+clean" 2026-02-20T16:34:21.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:21.978 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 0 == 4 ']' 2026-02-20T16:34:21.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:22.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:22.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:22.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:22.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:22.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:22.352 INFO:tasks.workunit.client.0.trial013.stdout:1 ) state now: "active+clean" 2026-02-20T16:34:22.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2026-02-20T16:34:22.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:22.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:22.353 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:22.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 1 == 4 ']' 2026-02-20T16:34:22.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:22.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:22.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:22.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:22.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:22.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:22.727 INFO:tasks.workunit.client.0.trial013.stdout:2 ) state now: "active+clean" 2026-02-20T16:34:22.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2026-02-20T16:34:22.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:22.728 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 2 == 4 ']' 2026-02-20T16:34:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:23.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:23.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:23.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:23.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:23.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:23.103 INFO:tasks.workunit.client.0.trial013.stdout:3 ) state now: "active+clean" 2026-02-20T16:34:23.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2026-02-20T16:34:23.104 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:23.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:23.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:23.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 3 == 4 ']' 2026-02-20T16:34:23.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:23.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:23.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:23.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:23.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:23.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:23.481 INFO:tasks.workunit.client.0.trial013.stdout:4 ) state now: "active+clean" 2026-02-20T16:34:23.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2026-02-20T16:34:23.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:23.482 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:23.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:23.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 4 == 4 ']' 2026-02-20T16:34:23.483 INFO:tasks.workunit.client.0.trial013.stdout:loop --------> 4 2026-02-20T16:34:23.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: initiate_and_fetch_state: echo 'loop --------> ' 4 2026-02-20T16:34:23.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:23.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:23.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:23.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:23.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:23.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2026-02-20T16:34:23.859 INFO:tasks.workunit.client.0.trial013.stdout:5 ) state now: "active+clean" 2026-02-20T16:34:23.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:23.860 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 5 == 4 ']' 2026-02-20T16:34:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:24.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:24.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:24.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:24.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:24.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:24.236 INFO:tasks.workunit.client.0.trial013.stdout:6 ) state now: "active+clean" 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 6 == 4 ']' 2026-02-20T16:34:24.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:24.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:24.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:24.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:24.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:24.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:24.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2026-02-20T16:34:24.612 INFO:tasks.workunit.client.0.trial013.stdout:7 ) state now: "active+clean" 2026-02-20T16:34:24.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:24.613 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:24.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:24.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 7 == 4 ']' 2026-02-20T16:34:24.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:24.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:24.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:24.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean"' 2026-02-20T16:34:24.986 INFO:tasks.workunit.client.0.trial013.stdout:8 ) state now: "active+clean" 2026-02-20T16:34:24.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:24.987 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:24.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:24.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 8 == 4 ']' 2026-02-20T16:34:24.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:25.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:25.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:25.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:25.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:25.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-20T16:34:25.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 9 ') state now: ' '"active+clean"' 2026-02-20T16:34:25.360 INFO:tasks.workunit.client.0.trial013.stdout:9 ) state now: "active+clean" 2026-02-20T16:34:25.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:25.361 INFO:tasks.workunit.client.0.trial013.stdout:"active+clean" 2026-02-20T16:34:25.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-20T16:34:25.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 9 == 4 ']' 2026-02-20T16:34:25.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-20T16:34:25.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-20T16:34:25.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-20T16:34:25.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-20T16:34:25.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-20T16:34:25.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:25.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 10 ') state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:25.735 INFO:tasks.workunit.client.0.trial013.stdout:10 ) state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:25.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-20T16:34:25.736 INFO:tasks.workunit.client.0.trial013.stdout:found scrub 2026-02-20T16:34:25.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: echo 'found scrub' 2026-02-20T16:34:25.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: return 0 2026-02-20T16:34:25.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_tag: r=0 2026-02-20T16:34:25.737 INFO:tasks.workunit.client.0.trial013.stdout:initiate_and_fetch_state ret: 0 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:34:25.739 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.td/osd-scrub-repair.asok 2026-02-20T16:34:25.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:485: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: wait_end_of_scrub: local the_osd=osd.1 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: wait_end_of_scrub: local pgid=1.0 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i=0 )) 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:25.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:25.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:25.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:25.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:25.854 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:25.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:25.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 0 == 4 ']' 2026-02-20T16:34:25.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-20T16:34:26.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-20T16:34:26.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:26.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:26.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:26.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.229 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:26.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:26.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 1 == 4 ']' 2026-02-20T16:34:26.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-20T16:34:26.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-20T16:34:26.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:26.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:26.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 2 == 4 ']' 2026-02-20T16:34:26.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-20T16:34:26.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-20T16:34:26.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:26.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:26.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:26.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.979 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:26.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:26.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:26.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 3 == 4 ']' 2026-02-20T16:34:26.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-20T16:34:27.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-20T16:34:27.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:27.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:27.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:27.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-20T16:34:27.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-20T16:34:27.356 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-20T16:34:27.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:27.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 4 == 4 ']' 2026-02-20T16:34:27.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:428: wait_end_of_scrub: flush_pg_stats 2026-02-20T16:34:27.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:27.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:27.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:27.512 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:27.512 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:27.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:27.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:27.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:27.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247814 2026-02-20T16:34:27.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247814 2026-02-20T16:34:27.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247814' 2026-02-20T16:34:27.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:27.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:27.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705673 2026-02-20T16:34:27.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705673 2026-02-20T16:34:27.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247814 1-38654705673' 2026-02-20T16:34:27.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:27.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574856 2026-02-20T16:34:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574856 2026-02-20T16:34:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247814 1-38654705673 2-55834574856' 2026-02-20T16:34:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:27.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247814 2026-02-20T16:34:27.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:27.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247814 2026-02-20T16:34:27.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:27.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247814 2026-02-20T16:34:27.733 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 98784247814 2026-02-20T16:34:27.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2026-02-20T16:34:27.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:27.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247814 2026-02-20T16:34:27.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:28.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:28.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:29.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247814 2026-02-20T16:34:29.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:30.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:34:30.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:30.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-20T16:34:30.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:30.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705673 2026-02-20T16:34:30.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:30.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:30.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705673 2026-02-20T16:34:30.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:30.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705673 2026-02-20T16:34:30.230 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705673 2026-02-20T16:34:30.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2026-02-20T16:34:30.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:30.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705673 2026-02-20T16:34:30.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:30.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574856 2026-02-20T16:34:30.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:30.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:30.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574856 2026-02-20T16:34:30.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:30.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574856 2026-02-20T16:34:30.395 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574856 2026-02-20T16:34:30.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574856' 2026-02-20T16:34:30.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:30.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574857 -lt 55834574856 2026-02-20T16:34:30.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-20T16:34:30.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-20T16:34:30.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-20T16:34:30.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-20T16:34:30.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-20T16:34:30.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean"' 2026-02-20T16:34:30.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2026-02-20T16:34:30.939 INFO:tasks.workunit.client.0.trial013.stdout:wait-scrub-end state now: "active+clean" 2026-02-20T16:34:30.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:30.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: break 2026-02-20T16:34:30.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-20T16:34:30.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: wait_end_of_scrub: return 0 2026-02-20T16:34:30.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:486: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2026-02-20T16:34:31.085 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:34:31.085 INFO:tasks.workunit.client.0.trial013.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-20T16:34:28.486409+0000 19'1 25:68 [1,0] 1 [1,0] 1 19'1 2026-02-20T16:34:28.484070+0000 19'1 2026-02-20T16:34:28.484070+0000 0 3 periodic scrub scheduled @ 2026-02-21T16:34:28.484070+0000 1 0 2026-02-20T16:34:31.085 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:34:31.085 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:34:31.085 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:34:31.095 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:31.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:34:31.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:34:31.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:31.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:34:31.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:34:31.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:34:31.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:34:31.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:34:31.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:34:31.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:34:31.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:34:31.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:34:31.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:34:31.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:34:31.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:34:31.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:31.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-20T16:34:31.796 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:34:31.796 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:34:31.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:31.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:31.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:31.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:31.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:34:31.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:34:31.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:34:31.891 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:34:31.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:31.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:34:31.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:34:31.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:34:31.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:34:31.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:34:31.914 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:31.944+0000 7ff02b79ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:31.916 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:31.944+0000 7ff02b79ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:31.916 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:31.944+0000 7ff02b79ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:32.048 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:34:32.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:32.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:32.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:32.331 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:32.360+0000 7ff02b79ca40 -1 Falling back to public interface 2026-02-20T16:34:32.947 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:32.976+0000 7ff02b79ca40 -1 osd.0 25 log_to_monitors true 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:33.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:33.371 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [23,25) [v2:127.0.0.1:6800/2897850595,v1:127.0.0.1:6801/2897850595] [v2:127.0.0.1:6802/2897850595,v1:127.0.0.1:6803/2897850595] exists,up 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:34:33.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:33.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:33.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:33.373 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:33.373 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:33.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:33.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:33.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:33.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:33.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:33.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:33.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:33.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:33.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:33.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:33.653 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:33.653 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:33.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:33.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:33.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:33.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-20T16:34:33.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-20T16:34:33.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-20T16:34:33.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:33.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:33.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705675 2026-02-20T16:34:33.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705675 2026-02-20T16:34:33.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-38654705675' 2026-02-20T16:34:33.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:33.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:33.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-20T16:34:33.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-20T16:34:33.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-38654705675 2-55834574859' 2026-02-20T16:34:33.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:33.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-20T16:34:33.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:33.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:33.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-20T16:34:33.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-20T16:34:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-20T16:34:33.872 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 124554051586 2026-02-20T16:34:33.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:34.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 124554051586 2026-02-20T16:34:34.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:35.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:35.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:35.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-20T16:34:35.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:35.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705675 2026-02-20T16:34:35.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:35.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:35.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705675 2026-02-20T16:34:35.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:35.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705675 2026-02-20T16:34:35.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-20T16:34:35.203 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705675 2026-02-20T16:34:35.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:35.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-20T16:34:35.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:35.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-20T16:34:35.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:35.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:35.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574859 2026-02-20T16:34:35.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:35.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574859 2026-02-20T16:34:35.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-20T16:34:35.376 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574859 2026-02-20T16:34:35.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:35.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-20T16:34:35.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:34:35.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:35.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:34:35.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:34:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:34:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:34:35.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:34:35.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:34:35.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:35.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:36.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:34:36.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:34:36.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:34:36.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-20T16:34:36.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:34:36.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:34:36.094 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-20T16:34:36.094 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:34:36.094 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:34:36.094 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:36.095 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:34:36.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:34:36.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-20T16:34:36.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:34:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:34:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:36.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:36.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:34:36.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:36.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:34:36.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:34:36.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-20T16:34:36.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:34:36.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:34:36.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:34:36.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:34:36.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:34:36.910 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:36.944+0000 7f44b9e05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:36.911 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:36.944+0000 7f44b9e05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:36.912 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:36.944+0000 7f44b9e05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:37.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:37.351 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:37.384+0000 7f44b9e05a40 -1 Falling back to public interface 2026-02-20T16:34:38.002 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:38.036+0000 7f44b9e05a40 -1 osd.0 30 log_to_monitors true 2026-02-20T16:34:38.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:38.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:38.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:38.206 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:38.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:38.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:38.361 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6800/545983684,v1:127.0.0.1:6801/545983684] [v2:127.0.0.1:6802/545983684,v1:127.0.0.1:6803/545983684] exists,up 0deb9c59-ac96-4ee0-8b78-84dea313f230 2026-02-20T16:34:38.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:38.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:38.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:38.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:34:38.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:38.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:38.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:38.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:38.363 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:38.363 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:38.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:38.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:38.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:38.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:38.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:38.637 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:38.637 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:38.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:38.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:38.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-20T16:34:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-20T16:34:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-20T16:34:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:38.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:38.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-20T16:34:38.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-20T16:34:38.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-38654705677' 2026-02-20T16:34:38.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:38.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:38.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574861 2026-02-20T16:34:38.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574861 2026-02-20T16:34:38.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-38654705677 2-55834574861' 2026-02-20T16:34:38.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:38.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-20T16:34:38.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:38.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:38.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-20T16:34:38.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:38.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-20T16:34:38.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-20T16:34:38.858 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 146028888066 2026-02-20T16:34:38.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:39.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 146028888066 2026-02-20T16:34:39.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:40.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:40.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:40.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-20T16:34:40.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:40.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-20T16:34:40.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:40.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:40.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-20T16:34:40.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:40.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-20T16:34:40.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-20T16:34:40.178 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705677 2026-02-20T16:34:40.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:40.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705677 -lt 38654705677 2026-02-20T16:34:40.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:40.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574861 2026-02-20T16:34:40.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:40.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:40.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574861 2026-02-20T16:34:40.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:40.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574861 2026-02-20T16:34:40.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574861' 2026-02-20T16:34:40.345 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574861 2026-02-20T16:34:40.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574861 -lt 55834574861 2026-02-20T16:34:40.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:34:40.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:40.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:40.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:34:40.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:34:40.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:34:40.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:34:40.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:34:40.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:40.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:34:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:34:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:34:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:34:41.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-20T16:34:41.043 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:34:28.481+0000 7fb53061f640 10 osd.1 pg_epoch: 25 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=14/14 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-20T16:34:41.043 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:34:28.481+0000 7fb53061f640 10 osd.1 pg_epoch: 25 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=14/14 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-20T16:34:41.043 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:34:28.481+0000 7fb53061f640 15 osd.1 pg_epoch: 25 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=14/14 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:34:28.481+0000 7fb53061f640 20 osd.1 pg_epoch: 25 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=14/14 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T16:34:28.481+0000 7fb53061f640 19 osd.1 pg_epoch: 25 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=14/14 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:34:41.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:34:41.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:34:41.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:34:41.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:34:41.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:34:41.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:34:41.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:34:41.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:34:41.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:34:41.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:34:41.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:34:41.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:34:41.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:34:41.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:34:41.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:34:41.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:34:41.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:34:41.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:34:41.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:34:41.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:34:41.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:34:41.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:34:41.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:34:41.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:34:41.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:34:41.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:34:41.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:34:41.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:34:41.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:34:41.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:34:41.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:34:41.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:34:41.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:34:41.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:34:41.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:34:41.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:34:41.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:34:41.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:34:41.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:34:41.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:34:41.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:34:41.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:34:41.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:34:41.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:34:41.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:34:41.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:34:41.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:34:41.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:34:41.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:34:41.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=false 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:34:41.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:34:41.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:34:41.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:34:41.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.214 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:41.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:34:41.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:34:41.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:34:41.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:34:41.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:34:41.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:34:41.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:34:41.248 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:34:41.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:34:41.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:34:41.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:34:41.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:34:41.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:34:41.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:34:41.307 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:34:41.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:34:41.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:34:41.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:34:41.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:41.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:41.459 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:41.460 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.460 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:41.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:34:41.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:34:41.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:34:41.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:34:41.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:41.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:41.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:41.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:41.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:41.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:34:41.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:34:41.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:34:41.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:34:41.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9c58ce79-1876-45fb-887f-f41fd07d32f0 2026-02-20T16:34:41.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9c58ce79-1876-45fb-887f-f41fd07d32f0' 2026-02-20T16:34:41.486 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 9c58ce79-1876-45fb-887f-f41fd07d32f0 2026-02-20T16:34:41.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:34:41.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhjZhpZ2COHxAA6pl83AGk5qWNOI6qrTrHjA== 2026-02-20T16:34:41.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhjZhpZ2COHxAA6pl83AGk5qWNOI6qrTrHjA=="}' 2026-02-20T16:34:41.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9c58ce79-1876-45fb-887f-f41fd07d32f0 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:34:41.583 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:41.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:34:41.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAhjZhpZ2COHxAA6pl83AGk5qWNOI6qrTrHjA== --osd-uuid 9c58ce79-1876-45fb-887f-f41fd07d32f0 2026-02-20T16:34:41.609 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:41.644+0000 7fe7b4d43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:41.611 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:41.644+0000 7fe7b4d43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:41.612 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:41.644+0000 7fe7b4d43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:41.612 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:41.644+0000 7fe7b4d43a40 -1 bdev(0x563a3e824c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:34:41.612 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:41.644+0000 7fe7b4d43a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:34:42.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:34:42.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:34:42.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:34:42.316 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:34:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:34:42.408 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:34:42.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:34:42.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:42.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:34:42.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:34:42.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:34:42.425 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:42.456+0000 7f30bd55aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:42.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:42.460+0000 7f30bd55aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:42.427 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:42.460+0000 7f30bd55aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:42.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:42.504 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:42.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:42.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:42.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:42.815 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:42.848+0000 7f30bd55aa40 -1 Falling back to public interface 2026-02-20T16:34:43.462 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:43.496+0000 7f30bd55aa40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:34:43.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:43.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:43.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:43.596 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:43.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:43.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:43.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:44.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:44.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:44.723 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:44.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:34:44.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:44.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:34:44.884 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3057306727,v1:127.0.0.1:6801/3057306727] [v2:127.0.0.1:6802/3057306727,v1:127.0.0.1:6803/3057306727] exists,up 9c58ce79-1876-45fb-887f-f41fd07d32f0 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:44.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:44.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:34:44.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 377f5647-ea34-46f0-84b1-1df404c42cc0' 2026-02-20T16:34:44.891 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:34:44.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:34:44.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkjZhpIKe2NxAACnwOW7IRpiO7lPDvOYgruQ== 2026-02-20T16:34:44.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkjZhpIKe2NxAACnwOW7IRpiO7lPDvOYgruQ=="}' 2026-02-20T16:34:44.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 377f5647-ea34-46f0-84b1-1df404c42cc0 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:34:45.063 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:45.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:34:45.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAkjZhpIKe2NxAACnwOW7IRpiO7lPDvOYgruQ== --osd-uuid 377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:34:45.091 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:45.124+0000 7ff2704a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:45.092 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:45.124+0000 7ff2704a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:45.093 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:45.128+0000 7ff2704a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:45.093 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:45.128+0000 7ff2704a3a40 -1 bdev(0x55de7ba5ec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:34:45.094 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:45.128+0000 7ff2704a3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:34:45.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:34:45.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:34:45.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:34:45.753 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:34:45.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:34:45.958 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:34:45.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:34:45.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:45.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:34:45.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:34:45.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:34:45.985 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:46.016+0000 7f82b486ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:45.987 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:46.020+0000 7f82b486ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:45.988 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:46.020+0000 7f82b486ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:46.123 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:46.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:46.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:46.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:34:46.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:46.395 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:46.428+0000 7f82b486ca40 -1 Falling back to public interface 2026-02-20T16:34:47.065 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:47.100+0000 7f82b486ca40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:34:47.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:47.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:47.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:47.286 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:47.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:47.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:34:47.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:48.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:48.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:48.442 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:48.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:34:48.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:48.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:34:48.595 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2405262813,v1:127.0.0.1:6811/2405262813] [v2:127.0.0.1:6812/2405262813,v1:127.0.0.1:6813/2405262813] exists,up 377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:34:48.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:48.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:48.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:34:48.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:34:48.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=74c41199-a308-41c2-ba01-0af0105e93e9 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 74c41199-a308-41c2-ba01-0af0105e93e9' 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 74c41199-a308-41c2-ba01-0af0105e93e9 2026-02-20T16:34:48.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:34:48.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAojZhp4hOSJhAAH/3v267ZvNGffcJS99jExw== 2026-02-20T16:34:48.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAojZhp4hOSJhAAH/3v267ZvNGffcJS99jExw=="}' 2026-02-20T16:34:48.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 74c41199-a308-41c2-ba01-0af0105e93e9 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:34:48.772 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:48.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:34:48.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAojZhp4hOSJhAAH/3v267ZvNGffcJS99jExw== --osd-uuid 74c41199-a308-41c2-ba01-0af0105e93e9 2026-02-20T16:34:48.800 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:48.832+0000 7fd737b28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:48.801 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:48.836+0000 7fd737b28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:48.802 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:48.836+0000 7fd737b28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:48.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:48.836+0000 7fd737b28a40 -1 bdev(0x5630108a6c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:34:48.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:48.836+0000 7fd737b28a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:34:49.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:34:49.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:34:49.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:34:49.429 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:34:49.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:34:49.626 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:34:49.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:34:49.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:34:49.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:34:49.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:34:49.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:34:49.644 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:49.676+0000 7feaec916a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:49.645 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:49.680+0000 7feaec916a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:49.646 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:49.680+0000 7feaec916a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:34:49.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:34:49.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:34:49.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:49.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:34:49.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:50.050 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:50.084+0000 7feaec916a40 -1 Falling back to public interface 2026-02-20T16:34:50.664 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:34:50.696+0000 7feaec916a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:34:50.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:50.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:50.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:34:50.947 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:34:50.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:50.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:34:51.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:34:52.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:34:52.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:34:52.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:34:52.109 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:34:52.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:34:52.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:34:52.263 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1324986056,v1:127.0.0.1:6819/1324986056] [v2:127.0.0.1:6820/1324986056,v1:127.0.0.1:6821/1324986056] exists,up 74c41199-a308-41c2-ba01-0af0105e93e9 2026-02-20T16:34:52.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:34:52.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:34:52.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:34:52.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-20T16:34:52.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:34:52.410 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:34:52.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:34:52.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:34:52.664 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:34:52.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:34:53.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:34:53.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-20T16:34:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:53.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:53.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:53.953 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:53.953 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:53.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:53.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:53.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:54.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:54.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:54.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:54.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:54.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:54.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:54.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:54.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:34:54.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:34:54.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:34:54.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:54.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:54.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:34:54.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:34:54.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-20T16:34:54.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:54.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:54.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-20T16:34:54.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-20T16:34:54.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-20T16:34:54.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:54.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:34:54.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:54.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:54.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:34:54.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:54.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:34:54.442 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:34:54.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:34:54.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:54.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:34:54.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:55.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:34:55.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:34:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:34:56.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:34:56.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:56.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:34:56.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:56.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:34:56.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:56.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:56.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:34:56.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:56.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:34:56.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:34:56.940 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:34:56.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:34:57.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:34:57.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:57.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-20T16:34:57.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:57.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:34:57.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-20T16:34:57.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:57.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-20T16:34:57.106 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574850 2026-02-20T16:34:57.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-20T16:34:57.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:34:57.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-20T16:34:57.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:34:57.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:57.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:57.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:34:57.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:34:57.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:34:57.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:34:57.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:34:57.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:34:57.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:34:57.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-20T16:34:58.020 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:34:58.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:34:58.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:34:58.326 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:34:58.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:34:59.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:34:59.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:34:59.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:34:59.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:34:59.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:34:59.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:34:59.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:34:59.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:34:59.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:34:59.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:34:59.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:34:59.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:59.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:34:59.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:34:59.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:34:59.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:34:59.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:59.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:34:59.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:34:59.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:34:59.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-20T16:34:59.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:34:59.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:34:59.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-20T16:34:59.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-20T16:34:59.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-55834574852' 2026-02-20T16:34:59.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:59.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:34:59.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:59.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:34:59.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:34:59.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:59.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:34:59.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:34:59.834 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:34:59.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:34:59.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:34:59.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:34:59.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:34:59.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:34:59.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:34:59.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:34:59.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:34:59.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:34:59.993 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:34:59.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:34:59.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:00.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:35:00.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:00.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-20T16:35:00.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:00.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:00.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-20T16:35:00.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:00.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-20T16:35:00.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-20T16:35:00.170 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574852 2026-02-20T16:35:00.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574852 2026-02-20T16:35:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:01.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:01.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574852 2026-02-20T16:35:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:02.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:35:02.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:02.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-20T16:35:02.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:02.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:02.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:02.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:02.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:02.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:03.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:03.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:03.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:03.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:03.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-20T16:35:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:35:03.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-20T16:35:03.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-20T16:35:03.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:35:03.226 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-20T16:35:03.226 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:35:03.226 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:35:03.226 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:03.226 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:35:03.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-20T16:35:03.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:03.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:35:03.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:35:03.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:35:03.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:35:03.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:35:03.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:35:03.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:35:03.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:35:04.045 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:04.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:04.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:04.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:04.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:35:04.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:04.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:35:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:35:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:35:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:35:04.163 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:04.196+0000 7f2688926a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:04.165 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:04.200+0000 7f2688926a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:04.166 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:04.200+0000 7f2688926a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:04.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:04.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:04.599 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:04.632+0000 7f2688926a40 -1 Falling back to public interface 2026-02-20T16:35:05.252 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:05.284+0000 7f2688926a40 -1 osd.1 27 log_to_monitors true 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:05.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [9,27) [v2:127.0.0.1:6810/2037332421,v1:127.0.0.1:6811/2037332421] [v2:127.0.0.1:6812/2037332421,v1:127.0.0.1:6813/2037332421] exists,up 377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:05.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:05.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:05.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:05.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:05.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:05.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:05.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:05.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:05.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:05.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:05.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:05.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:05.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:35:05.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:35:05.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:35:05.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:05.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:06.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-20T16:35:06.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-20T16:35:06.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-128849018882' 2026-02-20T16:35:06.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:06.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-20T16:35:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-20T16:35:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-128849018882 2-55834574854' 2026-02-20T16:35:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:06.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:35:06.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:06.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:06.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:35:06.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:06.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:35:06.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:35:06.123 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836488 2026-02-20T16:35:06.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:06.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:35:06.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:07.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:07.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:35:07.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:08.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:35:08.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:08.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:35:08.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:08.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018882 2026-02-20T16:35:08.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:08.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:08.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018882 2026-02-20T16:35:08.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:08.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-20T16:35:08.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2026-02-20T16:35:08.624 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 128849018882 2026-02-20T16:35:08.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:08.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-20T16:35:08.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:08.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574854 2026-02-20T16:35:08.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:08.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:08.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574854 2026-02-20T16:35:08.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:08.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574854 2026-02-20T16:35:08.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-20T16:35:08.799 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574854 2026-02-20T16:35:08.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:08.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574854 2026-02-20T16:35:08.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:08.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:08.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:09.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:09.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:09.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:09.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:09.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:09.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:09.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:09.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:09.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:09.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:09.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:09.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:35:09.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-20T16:35:09.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-20T16:35:09.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:09.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:09.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:09.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:09.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:09.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:09.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:03.505671+0000 '>' 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:09.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:10.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:10.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:10.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:10.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:10.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:10.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:10.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:11.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:03.505671+0000 '>' 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:11.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:12.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:12.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:12.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:12.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:12.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:12.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:12.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:12.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:03.505671+0000 '>' 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:12.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:13.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:13.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:13.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:13.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:13.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:13.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:13.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:03.505671+0000 '>' 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:13.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:14.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:14.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:14.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:14.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:14.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:14.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:14.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:14.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:11.464598+0000 '>' 2026-02-20T16:35:03.505671+0000 2026-02-20T16:35:14.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:35:14.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-20T16:35:14.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:14.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:14.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:14.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:14.627 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:14.627 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:14.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:14.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:14.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:14.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:14.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:14.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:14.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:14.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:14.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:14.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:14.900 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:14.900 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:14.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:14.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:14.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:14.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:35:14.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:35:14.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:35:14.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:14.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:15.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-20T16:35:15.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-20T16:35:15.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-128849018884' 2026-02-20T16:35:15.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:15.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-20T16:35:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-20T16:35:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-128849018884 2-55834574857' 2026-02-20T16:35:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:15.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:35:15.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:15.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:15.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:35:15.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:35:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:35:15.121 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T16:35:15.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:15.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-20T16:35:15.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:16.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:16.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:16.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:35:16.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:16.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018884 2026-02-20T16:35:16.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:16.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018884 2026-02-20T16:35:16.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:16.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-20T16:35:16.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018884' 2026-02-20T16:35:16.457 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 128849018884 2026-02-20T16:35:16.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018885 -lt 128849018884 2026-02-20T16:35:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:16.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-20T16:35:16.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:16.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:16.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-20T16:35:16.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:16.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-20T16:35:16.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-20T16:35:16.622 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574857 2026-02-20T16:35:16.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:16.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-20T16:35:16.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:16.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:16.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:16.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:16.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:16.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:16.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:16.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:16.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:16.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:16.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:17.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:17.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:17.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:17.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-20T16:35:17.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:35:17.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-20T16:35:17.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:35:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:35:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:17.325 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:35:17.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-20T16:35:17.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:17.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:35:17.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:35:17.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:35:18.024 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:35:18.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:18.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:35:18.139 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:35:18.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:18.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:35:18.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:35:18.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:35:18.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:35:18.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:35:18.159 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:18.192+0000 7fbd5d8faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:18.161 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:18.196+0000 7fbd5d8faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:18.162 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:18.196+0000 7fbd5d8faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:18.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:18.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:18.554 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:18.588+0000 7fbd5d8faa40 -1 Falling back to public interface 2026-02-20T16:35:19.186 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:19.220+0000 7fbd5d8faa40 -1 osd.1 32 log_to_monitors true 2026-02-20T16:35:19.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:19.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:19.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:19.455 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:19.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:19.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:19.616 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 33 last_clean_interval [30,32) [v2:127.0.0.1:6810/2898038119,v1:127.0.0.1:6811/2898038119] [v2:127.0.0.1:6812/2898038119,v1:127.0.0.1:6813/2898038119] exists,up 377f5647-ea34-46f0-84b1-1df404c42cc0 2026-02-20T16:35:19.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:19.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:19.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:19.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:35:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:19.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:19.618 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:19.618 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:19.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:19.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:19.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:19.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:19.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:19.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:19.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:19.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:19.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:19.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:19.881 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:19.881 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:19.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:19.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:19.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:19.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:35:19.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:35:19.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:35:19.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:19.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822658 2026-02-20T16:35:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822658 2026-02-20T16:35:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-154618822658' 2026-02-20T16:35:20.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:20.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:20.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-20T16:35:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-20T16:35:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-154618822658 2-55834574859' 2026-02-20T16:35:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:20.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:35:20.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:20.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:20.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:35:20.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:20.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:35:20.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:35:20.109 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:35:20.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:20.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T16:35:20.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:21.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:21.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:21.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T16:35:21.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:22.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:35:22.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:22.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T16:35:22.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:22.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-154618822658 2026-02-20T16:35:22.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:22.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:22.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-154618822658 2026-02-20T16:35:22.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822658 2026-02-20T16:35:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2026-02-20T16:35:22.599 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 154618822658 2026-02-20T16:35:22.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:22.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822658 -lt 154618822658 2026-02-20T16:35:22.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:22.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-20T16:35:22.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:22.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:22.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574859 2026-02-20T16:35:22.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:22.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574859 2026-02-20T16:35:22.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-20T16:35:22.767 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574859 2026-02-20T16:35:22.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:22.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-20T16:35:22.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:22.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:22.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:23.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:23.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:23.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:23.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:23.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:23.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:23.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:23.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:23.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:23.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:35:23.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:35:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:35:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:35:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:35:23.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:35:23.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:35:23.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:35:23.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:35:23.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:35:23.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:35:23.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:35:23.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:35:23.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:35:23.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:35:23.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:35:23.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:35:23.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:35:23.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:35:23.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:35:23.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:35:23.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:35:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:35:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:35:23.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:35:23.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:35:23.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:35:23.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:35:23.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:35:23.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:35:23.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:35:23.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:35:23.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:35:23.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:35:23.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:35:23.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:35:23.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:35:23.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:35:23.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:35:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:35:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:35:23.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:35:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:35:23.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:35:23.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:35:23.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:35:23.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:35:23.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:35:23.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:35:23.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:35:23.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:35:23.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:35:23.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=true 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:35:23.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:23.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:35:23.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:35:23.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:35:23.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:35:23.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:35:23.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:35:23.712 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.713 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.713 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:35:23.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:35:23.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:35:23.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:35:23.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:35:23.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:35:23.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:35:23.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:35:23.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:35:23.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:35:23.770 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:35:23.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:35:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:23.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:35:23.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:35:23.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:23.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:35:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:23.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:23.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:35:23.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:23.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:35:23.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:35:23.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=180517e1-f7f0-48ee-8937-e547b12eef03 2026-02-20T16:35:23.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 180517e1-f7f0-48ee-8937-e547b12eef03' 2026-02-20T16:35:23.949 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 180517e1-f7f0-48ee-8937-e547b12eef03 2026-02-20T16:35:23.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:35:23.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLjZhp5BcwOxAA9OiJGwiLjzqv4ZWO8a4vvA== 2026-02-20T16:35:23.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLjZhp5BcwOxAA9OiJGwiLjzqv4ZWO8a4vvA=="}' 2026-02-20T16:35:23.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 180517e1-f7f0-48ee-8937-e547b12eef03 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:35:24.048 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:35:24.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:35:24.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBLjZhp5BcwOxAA9OiJGwiLjzqv4ZWO8a4vvA== --osd-uuid 180517e1-f7f0-48ee-8937-e547b12eef03 2026-02-20T16:35:24.075 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:24.108+0000 7fc2040dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:24.077 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:24.108+0000 7fc2040dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:24.077 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:24.112+0000 7fc2040dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:24.078 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:24.112+0000 7fc2040dfa40 -1 bdev(0x55bb93000c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:35:24.078 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:24.112+0000 7fc2040dfa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:35:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:35:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:35:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:35:24.862 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:35:24.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:35:24.966 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:35:24.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:35:24.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:24.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:35:24.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:35:24.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:35:24.994 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:25.028+0000 7fea38041a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:24.996 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:25.028+0000 7fea38041a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:24.998 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:25.032+0000 7fea38041a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:25.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:35:25.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:35:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:25.063 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:35:25.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:25.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:25.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:35:25.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:25.527 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:25.560+0000 7fea38041a40 -1 Falling back to public interface 2026-02-20T16:35:26.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:26.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:26.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:26.157 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:35:26.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:26.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:35:26.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:26.458 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:26.492+0000 7fea38041a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:35:27.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:27.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:27.250 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:35:27.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:35:27.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:27.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:35:27.409 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2019435981,v1:127.0.0.1:6801/2019435981] [v2:127.0.0.1:6802/2019435981,v1:127.0.0.1:6803/2019435981] exists,up 180517e1-f7f0-48ee-8937-e547b12eef03 2026-02-20T16:35:27.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:27.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:27.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:27.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:27.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:27.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:27.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:35:27.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:27.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:35:27.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:35:27.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:35:27.418 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:35:27.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 744f6236-60b8-4886-b55f-e91c476c4eac' 2026-02-20T16:35:27.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:35:27.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBPjZhp/Gt6GxAAKxb+QuAD8nJHqPC9j/CwjA== 2026-02-20T16:35:27.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBPjZhp/Gt6GxAAKxb+QuAD8nJHqPC9j/CwjA=="}' 2026-02-20T16:35:27.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 744f6236-60b8-4886-b55f-e91c476c4eac -i td/osd-scrub-repair/1/new.json 2026-02-20T16:35:27.594 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:35:27.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:35:27.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBPjZhp/Gt6GxAAKxb+QuAD8nJHqPC9j/CwjA== --osd-uuid 744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:35:27.621 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:27.652+0000 7f60f12b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:27.622 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:27.656+0000 7f60f12b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:27.623 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:27.656+0000 7f60f12b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:27.624 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:27.656+0000 7f60f12b4a40 -1 bdev(0x564c4ab10c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:35:27.624 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:27.656+0000 7f60f12b4a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:35:28.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:35:28.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:35:28.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:35:28.282 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:35:28.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:35:28.486 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:35:28.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:35:28.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:28.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:35:28.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:35:28.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:35:28.504 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:28.535+0000 7fad702f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:28.505 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:28.539+0000 7fad702f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:28.506 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:28.539+0000 7fad702f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:35:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:35:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:28.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:28.642 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:35:28.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:28.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:28.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:28.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:28.923 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:28.955+0000 7fad702f8a40 -1 Falling back to public interface 2026-02-20T16:35:29.579 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:29.611+0000 7fad702f8a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:35:29.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:29.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:29.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:29.803 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:35:29.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:29.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:29.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:30.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:31.120 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1321076399,v1:127.0.0.1:6811/1321076399] [v2:127.0.0.1:6812/1321076399,v1:127.0.0.1:6813/1321076399] exists,up 744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:35:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:31.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:35:31.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:35:31.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cf976ef0-3e08-448f-a419-0db90f497688 2026-02-20T16:35:31.125 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 cf976ef0-3e08-448f-a419-0db90f497688 2026-02-20T16:35:31.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 cf976ef0-3e08-448f-a419-0db90f497688' 2026-02-20T16:35:31.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:35:31.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTjZhp+glUChAALRSUdTr6fw4CZdnoU+EhBQ== 2026-02-20T16:35:31.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTjZhp+glUChAALRSUdTr6fw4CZdnoU+EhBQ=="}' 2026-02-20T16:35:31.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cf976ef0-3e08-448f-a419-0db90f497688 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:35:31.299 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:35:31.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:35:31.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBTjZhp+glUChAALRSUdTr6fw4CZdnoU+EhBQ== --osd-uuid cf976ef0-3e08-448f-a419-0db90f497688 2026-02-20T16:35:31.327 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:31.359+0000 7fe5f6ea3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:31.329 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:31.363+0000 7fe5f6ea3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:31.330 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:31.363+0000 7fe5f6ea3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:31.330 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:31.363+0000 7fe5f6ea3a40 -1 bdev(0x55f87db16c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:35:31.330 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:31.363+0000 7fe5f6ea3a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:35:31.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:35:31.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:35:31.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:35:31.961 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:35:31.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:35:32.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:35:32.184 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:32.215+0000 7f08f8488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:32.186 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:32.219+0000 7f08f8488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:32.187 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:32.219+0000 7f08f8488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:32.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:32.317 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:35:32.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:32.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:32.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:35:32.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:32.619 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:32.651+0000 7f08f8488a40 -1 Falling back to public interface 2026-02-20T16:35:33.239 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:33.271+0000 7f08f8488a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:35:33.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:33.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:33.479 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:35:33.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:33.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:33.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:35:33.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:34.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:34.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:34.644 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:35:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:35:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:35:34.806 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2260697878,v1:127.0.0.1:6819/2260697878] [v2:127.0.0.1:6820/2260697878,v1:127.0.0.1:6821/2260697878] exists,up cf976ef0-3e08-448f-a419-0db90f497688 2026-02-20T16:35:34.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-20T16:35:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:35:34.957 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:35:34.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:35:34.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:35:35.152 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:35:35.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:35:36.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:35:36.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-20T16:35:36.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:36.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:36.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:36.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:36.446 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:36.446 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:36.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:36.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:36.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:36.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:36.714 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:36.714 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:36.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:36.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:36.715 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:36.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:35:36.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:35:36.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:35:36.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:36.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:36.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:35:36.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:35:36.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-20T16:35:36.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:36.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:36.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-20T16:35:36.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-20T16:35:36.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-60129542146' 2026-02-20T16:35:36.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:36.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:35:36.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:36.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:36.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:35:36.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:36.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:35:36.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:35:36.935 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:35:36.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:37.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:35:37.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:38.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:38.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:38.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:35:38.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:38.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:35:38.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:38.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:38.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:35:38.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:38.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:35:38.270 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:35:38.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:35:38.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:38.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:35:38.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:38.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-20T16:35:38.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:38.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:38.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-20T16:35:38.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:38.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-20T16:35:38.446 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542146 2026-02-20T16:35:38.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-20T16:35:38.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-20T16:35:38.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:38.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:38.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:38.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:35:38.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:38.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:35:38.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:38.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:38.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:39.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:35:39.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-20T16:35:39.385 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:35:39.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:35:39.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:35:39.693 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:35:39.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:35:40.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T16:35:40.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T16:35:40.907 INFO:tasks.workunit.client.0.trial013.stderr:set pool 2 allow_ec_overwrites to true 2026-02-20T16:35:40.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:35:40.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:40.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:40.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:40.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:40.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:40.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:40.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:40.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:40.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:41.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:41.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:41.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:41.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:41.195 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:41.195 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:41.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:41.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:41.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:41.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:35:41.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:35:41.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:35:41.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:41.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:41.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:35:41.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:35:41.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-20T16:35:41.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:41.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:41.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T16:35:41.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T16:35:41.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-60129542148' 2026-02-20T16:35:41.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:41.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:35:41.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:41.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:41.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:35:41.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:41.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:35:41.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:35:41.417 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:35:41.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:41.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:35:41.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:42.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:42.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:42.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:35:42.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:42.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:35:42.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:42.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:42.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:35:42.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:42.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:35:42.742 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:35:42.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:35:42.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:42.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:35:42.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:42.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T16:35:42.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:42.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:42.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T16:35:42.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:42.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T16:35:42.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T16:35:42.906 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T16:35:42.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:43.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-20T16:35:43.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:43.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:43.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:43.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:43.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:43.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:43.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:43.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:43.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:43.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:43.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-20T16:35:43.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:35:43.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-20T16:35:43.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-20T16:35:43.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:35:43.638 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-20T16:35:43.638 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:35:43.638 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:35:43.638 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:43.638 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:35:43.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-20T16:35:43.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:43.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:35:43.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:35:43.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:35:43.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:35:43.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:35:43.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:35:43.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:35:44.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:35:44.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:35:44.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:35:44.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:35:44.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:35:44.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:35:44.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:44.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:35:44.480 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-20T16:35:44.481 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-20T16:35:44.481 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:35:44.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:35:44.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:35:44.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:35:44.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:35:44.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:35:44.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:35:44.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:44.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:35:44.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:35:44.597 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:35:44.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:35:44.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:35:44.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:35:44.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:35:44.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:35:44.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:35:44.619 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:44.651+0000 7fa21290ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:44.621 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:44.651+0000 7fa21290ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:44.621 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:44.655+0000 7fa21290ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:35:44.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:35:44.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:35:44.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:35:44.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:35:44.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:35:44.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:44.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:35:44.758 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:35:44.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:44.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:44.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:35:44.995 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:45.027+0000 7fa21290ea40 -1 Falling back to public interface 2026-02-20T16:35:45.600 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:35:45.631+0000 7fa21290ea40 -1 osd.1 28 log_to_monitors true 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:35:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:35:46.076 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [9,28) [v2:127.0.0.1:6810/3200356332,v1:127.0.0.1:6811/3200356332] [v2:127.0.0.1:6812/3200356332,v1:127.0.0.1:6813/3200356332] exists,up 744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:35:46.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:35:46.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:35:46.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:35:46.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:35:46.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:46.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:46.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:46.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:46.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:46.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:46.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:46.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:46.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:46.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:46.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:46.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:46.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:46.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:46.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:46.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:46.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-20T16:35:46.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-20T16:35:46.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-20T16:35:46.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:46.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:46.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953474 2026-02-20T16:35:46.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953474 2026-02-20T16:35:46.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-137438953474' 2026-02-20T16:35:46.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:46.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:46.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-20T16:35:46.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-20T16:35:46.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-137438953474 2-60129542150' 2026-02-20T16:35:46.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:46.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-20T16:35:46.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:46.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:46.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-20T16:35:46.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:46.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-20T16:35:46.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-20T16:35:46.552 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836487 2026-02-20T16:35:46.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:46.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:35:46.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:47.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:47.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:47.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:35:47.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:48.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:35:48.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:49.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-20T16:35:49.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:49.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-137438953474 2026-02-20T16:35:49.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:49.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:49.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-137438953474 2026-02-20T16:35:49.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:49.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953474 2026-02-20T16:35:49.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2026-02-20T16:35:49.048 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 137438953474 2026-02-20T16:35:49.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:49.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953474 -lt 137438953474 2026-02-20T16:35:49.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:49.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-20T16:35:49.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:49.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:49.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-20T16:35:49.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:49.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-20T16:35:49.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-20T16:35:49.217 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542150 2026-02-20T16:35:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:49.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-20T16:35:49.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:49.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:49.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:49.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:49.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:49.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:49.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:49.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:49.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:49.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:35:49.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:35:50.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-20T16:35:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-20T16:35:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:50.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:50.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:50.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:50.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:50.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:50.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:48.176566+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:50.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:51.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:51.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:51.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:51.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:51.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:51.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:51.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:51.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:48.176566+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:51.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:52.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:52.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:52.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:52.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:52.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:52.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:52.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:52.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:48.176566+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:52.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:53.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:53.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:53.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:53.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:53.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:53.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:53.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:53.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:48.176566+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:53.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:54.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:54.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:54.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:54.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:54.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:54.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:54.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:55.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:48.176566+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:55.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:35:56.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:35:56.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:35:56.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:35:56.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:35:56.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:35:56.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:35:56.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:35:56.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:35:54.136789+0000 '>' 2026-02-20T16:35:48.176566+0000 2026-02-20T16:35:56.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:35:56.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-20T16:35:56.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:35:56.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:35:56.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:35:56.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:35:56.213 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:35:56.214 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:35:56.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:35:56.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:35:56.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:35:56.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:35:56.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:35:56.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:35:56.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:35:56.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:35:56.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:35:56.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:35:56.484 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:35:56.484 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:35:56.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:35:56.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:56.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:35:56.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-20T16:35:56.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-20T16:35:56.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-20T16:35:56.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:56.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:35:56.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953477 2026-02-20T16:35:56.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953477 2026-02-20T16:35:56.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-137438953477' 2026-02-20T16:35:56.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:35:56.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:35:56.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-20T16:35:56.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-20T16:35:56.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-137438953477 2-60129542153' 2026-02-20T16:35:56.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:56.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-20T16:35:56.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:56.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:35:56.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-20T16:35:56.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:56.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-20T16:35:56.709 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T16:35:56.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-20T16:35:56.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:56.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T16:35:56.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:57.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:35:57.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:58.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T16:35:58.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:35:59.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:35:59.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:35:59.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-20T16:35:59.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:59.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-137438953477 2026-02-20T16:35:59.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:59.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:35:59.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-137438953477 2026-02-20T16:35:59.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:59.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953477 2026-02-20T16:35:59.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 137438953477' 2026-02-20T16:35:59.187 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 137438953477 2026-02-20T16:35:59.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:35:59.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953477 -lt 137438953477 2026-02-20T16:35:59.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:35:59.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-20T16:35:59.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:35:59.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:35:59.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-20T16:35:59.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:35:59.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-20T16:35:59.358 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542153 2026-02-20T16:35:59.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-20T16:35:59.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:35:59.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-20T16:35:59.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:35:59.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:59.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:35:59.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:35:59.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:35:59.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:35:59.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:35:59.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:35:59.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:35:59.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-20T16:36:00.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:36:00.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-20T16:36:00.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:36:00.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:36:00.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:00.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:36:00.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-20T16:36:00.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:00.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:00.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:00.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:00.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:36:00.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:36:00.793 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:36:00.794 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:36:00.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:00.897 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:36:00.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:36:00.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:36:00.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:36:00.900 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:36:00.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:36:00.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:36:00.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:36:00.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:36:00.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:36:00.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:36:00.917 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:00.951+0000 7fe3dc1d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:00.918 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:00.951+0000 7fe3dc1d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:00.919 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:00.951+0000 7fe3dc1d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:01.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:36:01.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:01.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:36:01.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:01.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:01.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:01.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:01.058 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:36:01.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:01.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:01.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:01.330 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:01.363+0000 7fe3dc1d3a40 -1 Falling back to public interface 2026-02-20T16:36:01.953 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:01.983+0000 7fe3dc1d3a40 -1 osd.1 35 log_to_monitors true 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:02.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:02.381 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [32,35) [v2:127.0.0.1:6810/2286693775,v1:127.0.0.1:6811/2286693775] [v2:127.0.0.1:6812/2286693775,v1:127.0.0.1:6813/2286693775] exists,up 744f6236-60b8-4886-b55f-e91c476c4eac 2026-02-20T16:36:02.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:02.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:02.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:02.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:36:02.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:02.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:02.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:02.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:02.383 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:02.383 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:02.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:02.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:02.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:02.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:02.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:02.657 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:02.658 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:36:02.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:02.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:02.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:02.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:36:02.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:36:02.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:36:02.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:02.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:02.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-20T16:36:02.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-20T16:36:02.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-167503724546' 2026-02-20T16:36:02.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:02.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:02.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-20T16:36:02.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-20T16:36:02.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-167503724546 2-60129542155' 2026-02-20T16:36:02.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:02.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:36:02.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:02.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:02.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:36:02.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:02.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:36:02.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:36:02.881 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:36:02.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:03.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T16:36:03.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:04.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:04.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:04.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T16:36:04.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:05.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:36:05.209 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:05.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T16:36:05.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:05.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724546 2026-02-20T16:36:05.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:05.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:05.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724546 2026-02-20T16:36:05.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:05.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-20T16:36:05.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2026-02-20T16:36:05.374 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 167503724546 2026-02-20T16:36:05.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:05.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-20T16:36:05.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:05.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-20T16:36:05.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:05.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:05.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-20T16:36:05.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:05.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-20T16:36:05.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-20T16:36:05.539 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542155 2026-02-20T16:36:05.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:05.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-20T16:36:05.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:05.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:05.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:05.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:05.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:05.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:06.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:36:06.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:06.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:06.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:36:06.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:06.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:06.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:06.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:06.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:06.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:06.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:36:06.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:36:06.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:36:06.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:36:06.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:36:06.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:36:06.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:36:06.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:36:06.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:36:06.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:36:06.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:36:06.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:36:06.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:36:06.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:36:06.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:36:06.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:06.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:06.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:36:06.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:36:06.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:36:06.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:36:06.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:36:06.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:36:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:36:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:36:06.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:36:06.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:36:06.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:36:06.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:36:06.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:36:06.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:36:06.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:36:06.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:36:06.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:36:06.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:36:06.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:36:06.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:36:06.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:874: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=false 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-20T16:36:06.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:36:06.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:36:06.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:36:06.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:36:06.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:36:06.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:36:06.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:06.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:06.453 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:06.453 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.453 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:06.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:36:06.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:36:06.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:36:06.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:36:06.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:36:06.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:36:06.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:36:06.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:36:06.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:36:06.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:36:06.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:36:06.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:36:06.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:36:06.552 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:36:06.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:36:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:36:06.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:06.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:36:06.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:36:06.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-20T16:36:06.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:36:06.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:36:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:36:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:36:06.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:36:06.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1500cdd2-4cda-41ae-b920-daac8c157941 2026-02-20T16:36:06.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1500cdd2-4cda-41ae-b920-daac8c157941' 2026-02-20T16:36:06.732 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 1500cdd2-4cda-41ae-b920-daac8c157941 2026-02-20T16:36:06.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:36:06.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2jZhppqoNLBAA4gvjj1KKhzvufJioWwSwSA== 2026-02-20T16:36:06.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2jZhppqoNLBAA4gvjj1KKhzvufJioWwSwSA=="}' 2026-02-20T16:36:06.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1500cdd2-4cda-41ae-b920-daac8c157941 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:36:06.826 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:36:06.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:36:06.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2jZhppqoNLBAA4gvjj1KKhzvufJioWwSwSA== --osd-uuid 1500cdd2-4cda-41ae-b920-daac8c157941 2026-02-20T16:36:06.850 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:06.845+0000 7f65be473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:06.852 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:06.845+0000 7f65be473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:06.853 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:06.849+0000 7f65be473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:06.854 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:06.849+0000 7f65be473a40 -1 bdev(0x55d4268c6c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:36:06.854 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:06.849+0000 7f65be473a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:36:07.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:36:07.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:36:07.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:36:07.635 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:36:07.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:36:07.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:36:07.742 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:36:07.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:07.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:36:07.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:36:07.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:36:07.766 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:07.761+0000 7f91712dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:07.768 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:07.761+0000 7f91712dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:07.770 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:07.765+0000 7f91712dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:07.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:07.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:07.839 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:36:07.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:07.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:36:07.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:08.351 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:08.345+0000 7f91712dea40 -1 Falling back to public interface 2026-02-20T16:36:08.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:08.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:08.932 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:36:08.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:08.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:08.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:36:09.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:09.294 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:09.289+0000 7f91712dea40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:36:10.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:10.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:10.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:36:10.027 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:36:10.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:10.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:36:10.184 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1683279792,v1:127.0.0.1:6801/1683279792] [v2:127.0.0.1:6802/1683279792,v1:127.0.0.1:6803/1683279792] exists,up 1500cdd2-4cda-41ae-b920-daac8c157941 2026-02-20T16:36:10.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:10.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:10.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:10.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:10.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:10.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:10.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:36:10.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:36:10.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:36:10.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:36:10.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:36:10.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 840f1ec4-9418-4c4f-af60-4abb65283728' 2026-02-20T16:36:10.193 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:36:10.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:36:10.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6jZhpGOD5CxAAkiw399BaN1na6bkCQZpAJg== 2026-02-20T16:36:10.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6jZhpGOD5CxAAkiw399BaN1na6bkCQZpAJg=="}' 2026-02-20T16:36:10.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 840f1ec4-9418-4c4f-af60-4abb65283728 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:36:10.368 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:36:10.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:36:10.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6jZhpGOD5CxAAkiw399BaN1na6bkCQZpAJg== --osd-uuid 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:36:10.395 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:10.388+0000 7f4c134eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:10.396 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:10.388+0000 7f4c134eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:10.397 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:10.392+0000 7f4c134eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:10.398 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:10.392+0000 7f4c134eca40 -1 bdev(0x55c398010c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:36:10.398 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:10.392+0000 7f4c134eca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:36:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:36:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:36:11.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:36:11.085 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:36:11.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:36:11.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:36:11.284 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:36:11.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:11.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:36:11.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:36:11.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:36:11.309 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:11.300+0000 7f9892277a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:11.424 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:11.304+0000 7f9892277a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:11.424 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:11.304+0000 7f9892277a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:11.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:36:11.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:11.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:36:11.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:11.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:11.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:11.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:11.439 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:36:11.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:11.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:11.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:11.710 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:11.704+0000 7f9892277a40 -1 Falling back to public interface 2026-02-20T16:36:12.276 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:12.268+0000 7f9892277a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:36:12.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:12.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:12.600 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:36:12.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:12.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:12.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:12.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:13.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:13.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:13.761 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:36:13.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:36:13.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:13.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:13.921 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1056109039,v1:127.0.0.1:6811/1056109039] [v2:127.0.0.1:6812/1056109039,v1:127.0.0.1:6813/1056109039] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:36:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:36:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:36:13.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:13.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fb47810d-e5c7-4020-88ad-2a2eac97d9f6' 2026-02-20T16:36:13.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:36:13.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9jZhpRmXUNxAAHozuE1tbAK96TlaQESzTuQ== 2026-02-20T16:36:13.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9jZhpRmXUNxAAHozuE1tbAK96TlaQESzTuQ=="}' 2026-02-20T16:36:13.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fb47810d-e5c7-4020-88ad-2a2eac97d9f6 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:36:14.101 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:36:14.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:36:14.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9jZhpRmXUNxAAHozuE1tbAK96TlaQESzTuQ== --osd-uuid fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:36:14.129 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.124+0000 7f69493f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:14.130 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.124+0000 7f69493f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:14.131 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.124+0000 7f69493f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:14.132 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.124+0000 7f69493f4a40 -1 bdev(0x55e1a7a3ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:36:14.132 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.124+0000 7f69493f4a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:36:14.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:36:14.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:36:14.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:36:14.765 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:36:14.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:36:14.963 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:36:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:36:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:36:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:36:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:36:14.987 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.980+0000 7fa548238a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:14.989 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.984+0000 7fa548238a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:14.990 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:14.984+0000 7fa548238a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:15.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:36:15.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:36:15.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:15.362 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:15.356+0000 7fa548238a40 -1 Falling back to public interface 2026-02-20T16:36:16.067 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:16.060+0000 7fa548238a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:36:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:16.283 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:36:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:36:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:17.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:17.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:17.443 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:36:17.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:36:17.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:17.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:36:17.601 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/523639282,v1:127.0.0.1:6819/523639282] [v2:127.0.0.1:6820/523639282,v1:127.0.0.1:6821/523639282] exists,up fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:36:17.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:17.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:17.606 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:17.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:17.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:36:17.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:36:17.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:36:17.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:36:17.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:17.609 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:17.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e8334c2d-4a4c-4b81-9b53-6101f9f010cb' 2026-02-20T16:36:17.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:36:17.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBjZhpM1bGJBAAQmRftNSV1fEXqQLN3T2zPQ== 2026-02-20T16:36:17.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBjZhpM1bGJBAAQmRftNSV1fEXqQLN3T2zPQ=="}' 2026-02-20T16:36:17.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e8334c2d-4a4c-4b81-9b53-6101f9f010cb -i td/osd-scrub-repair/3/new.json 2026-02-20T16:36:17.781 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T16:36:17.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T16:36:17.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBjZhpM1bGJBAAQmRftNSV1fEXqQLN3T2zPQ== --osd-uuid e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:17.807 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:17.800+0000 7ff658e7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:17.808 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:17.800+0000 7ff658e7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:17.809 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:17.804+0000 7ff658e7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:17.810 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:17.804+0000 7ff658e7fa40 -1 bdev(0x55c9b0fa4c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T16:36:17.810 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:17.804+0000 7ff658e7fa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T16:36:18.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T16:36:18.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:36:18.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T16:36:18.459 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T16:36:18.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:36:18.660 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T16:36:18.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T16:36:18.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:18.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:36:18.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:36:18.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:36:18.676 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:18.668+0000 7f6f70765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:18.677 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:18.672+0000 7f6f70765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:18.678 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:18.672+0000 7f6f70765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:18.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:18.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:18.815 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:36:18.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:18.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:18.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:19.110 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:19.104+0000 7f6f70765a40 -1 Falling back to public interface 2026-02-20T16:36:19.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:19.688+0000 7f6f70765a40 -1 osd.3 0 log_to_monitors true 2026-02-20T16:36:19.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:19.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:19.980 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:36:19.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:19.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:19.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:20.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:21.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:21.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:21.138 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:36:21.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:36:21.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:21.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4053823575,v1:127.0.0.1:6827/4053823575] [v2:127.0.0.1:6828/4053823575,v1:127.0.0.1:6829/4053823575] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-20T16:36:21.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:36:21.446 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:36:21.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:36:21.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:36:21.662 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:36:21.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:36:22.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:36:22.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-20T16:36:22.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:22.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:22.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:22.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:22.957 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:22.957 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:22.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:22.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:22.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:23.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:23.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:23.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:23.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:23.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:23.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:23.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:23.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T16:36:23.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T16:36:23.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T16:36:23.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:23.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:23.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:36:23.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:36:23.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-20T16:36:23.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:23.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:23.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-20T16:36:23.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-20T16:36:23.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-20T16:36:23.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:23.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:36:23.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345922 2026-02-20T16:36:23.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345922 2026-02-20T16:36:23.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443 3-85899345922' 2026-02-20T16:36:23.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:23.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T16:36:23.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:23.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:23.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T16:36:23.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:23.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T16:36:23.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T16:36:23.520 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T16:36:23.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:23.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:36:23.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:24.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:24.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:24.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:36:24.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:25.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:36:25.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:26.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T16:36:26.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:26.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:36:26.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:26.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:26.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:36:26.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:26.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:36:26.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:36:26.017 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:36:26.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:26.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-20T16:36:26.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:26.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-20T16:36:26.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:26.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:26.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-20T16:36:26.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:26.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-20T16:36:26.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-20T16:36:26.183 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509443 2026-02-20T16:36:26.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:26.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-20T16:36:26.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:26.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-85899345922 2026-02-20T16:36:26.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:26.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:36:26.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-85899345922 2026-02-20T16:36:26.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:26.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345922 2026-02-20T16:36:26.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2026-02-20T16:36:26.362 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 85899345922 2026-02-20T16:36:26.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:36:26.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345922 2026-02-20T16:36:26.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:26.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:26.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:36:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:26.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:36:26.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:26.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:26.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:36:27.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-20T16:36:27.257 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:36:27.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:36:27.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:36:27.563 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:36:27.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:36:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:36:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:36:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:28.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:28.576 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:28.576 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:28.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:28.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:28.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:28.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:28.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-20T16:36:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-20T16:36:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-20T16:36:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:28.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:28.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-20T16:36:28.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-20T16:36:28.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2026-02-20T16:36:28.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:28.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:29.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509445 2026-02-20T16:36:29.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509445 2026-02-20T16:36:29.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509445' 2026-02-20T16:36:29.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:29.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:36:29.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345924 2026-02-20T16:36:29.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345924 2026-02-20T16:36:29.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509445 3-85899345924' 2026-02-20T16:36:29.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:29.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-20T16:36:29.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:29.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:29.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-20T16:36:29.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:29.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-20T16:36:29.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-20T16:36:29.141 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836486 2026-02-20T16:36:29.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:29.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-20T16:36:29.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:29.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-20T16:36:29.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:29.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:29.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-20T16:36:29.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:29.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-20T16:36:29.311 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672966 2026-02-20T16:36:29.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-20T16:36:29.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:29.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672966 2026-02-20T16:36:29.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:30.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:30.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:30.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672966 2026-02-20T16:36:30.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:31.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:36:31.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-20T16:36:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:31.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509445 2026-02-20T16:36:31.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:31.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:31.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509445 2026-02-20T16:36:31.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:31.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509445 2026-02-20T16:36:31.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-20T16:36:31.806 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509445 2026-02-20T16:36:31.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:31.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-20T16:36:31.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:31.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-85899345924 2026-02-20T16:36:31.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:31.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:36:31.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-85899345924 2026-02-20T16:36:31.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:31.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345924 2026-02-20T16:36:31.980 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 85899345924 2026-02-20T16:36:31.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2026-02-20T16:36:31.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:36:32.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345925 -lt 85899345924 2026-02-20T16:36:32.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:32.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:32.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:32.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:32.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:32.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:32.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:36:32.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:32.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:32.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-20T16:36:32.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:36:32.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:36:32.890 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:36:32.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:36:33.096 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:36:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:36:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:36:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:36:33.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T16:36:33.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:36:33.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:36:33.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:33.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:36:33.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-20T16:36:33.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T16:36:33.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:36:33.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T16:36:33.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-20T16:36:33.296 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:33.296 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:36:33.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-20T16:36:33.458 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:33.458 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:33.458 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:36:33.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-20T16:36:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:36:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:36:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:33.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:36:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:36:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:36:33.969 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:36:34.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:34.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:34.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:36:34.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:36:34.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:36:34.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:36:34.083 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:34.076+0000 7f024c6a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:34.085 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:34.080+0000 7f024c6a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:34.086 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:34.080+0000 7f024c6a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:34.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:34.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:34.220 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:36:34.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:34.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:34.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:34.494 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:34.488+0000 7f024c6a7a40 -1 Falling back to public interface 2026-02-20T16:36:35.121 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:35.116+0000 7f024c6a7a40 -1 osd.3 34 log_to_monitors true 2026-02-20T16:36:35.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:35.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:35.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:35.377 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:35.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:35.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:35.540 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/3142536030,v1:127.0.0.1:6827/3142536030] [v2:127.0.0.1:6828/3142536030,v1:127.0.0.1:6829/3142536030] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:35.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:35.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:35.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:35.541 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:35.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:35.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:35.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:35.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:35.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:35.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:35.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:35.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:35.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:35.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:35.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-20T16:36:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-20T16:36:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-20T16:36:35.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:35.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:35.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-20T16:36:35.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-20T16:36:35.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-20T16:36:35.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:35.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-20T16:36:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-20T16:36:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2026-02-20T16:36:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:36.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:36:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-20T16:36:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-20T16:36:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-163208757250' 2026-02-20T16:36:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:36.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-20T16:36:36.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:36.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:36.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-20T16:36:36.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:36.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-20T16:36:36.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-20T16:36:36.100 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836489 2026-02-20T16:36:36.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:36.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-20T16:36:36.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:37.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:37.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:37.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-20T16:36:37.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:37.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-20T16:36:37.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:37.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:37.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-20T16:36:37.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:37.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-20T16:36:37.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-20T16:36:37.432 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672968 2026-02-20T16:36:37.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:37.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-20T16:36:37.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:37.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-20T16:36:37.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:37.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:37.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-20T16:36:37.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:37.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-20T16:36:37.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-20T16:36:37.606 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509447 2026-02-20T16:36:37.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:37.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-20T16:36:37.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:37.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-163208757250 2026-02-20T16:36:37.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:37.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:36:37.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-163208757250 2026-02-20T16:36:37.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:37.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-20T16:36:37.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 163208757250' 2026-02-20T16:36:37.781 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 163208757250 2026-02-20T16:36:37.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:36:37.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-20T16:36:37.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:37.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:37.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:38.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:36:38.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:38.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:38.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:36:38.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:38.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:38.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:38.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:38.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:38.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:38.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:38.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:38.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:38.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:36:38.957 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:38.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:38.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:38.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:39.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:27.561217+0000 '>' 2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:39.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:40.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:40.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:40.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:27.561217+0000 '>' 2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:40.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:41.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:41.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:41.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:27.561217+0000 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:36:41.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:36:41.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:41.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:41.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:36:41.831 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:36:41.831 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:36:41.831 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:36:41.831 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:36:41.831 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:36:41.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:41.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:36:41.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:36:41.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:36:41.923 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:36:41.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:41.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:36:41.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:36:41.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:36:41.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:36:41.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:36:41.940 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:41.936+0000 7f175681fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:41.942 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:41.936+0000 7f175681fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:41.943 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:41.936+0000 7f175681fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:42.085 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:36:42.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:42.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:42.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:42.394 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:42.388+0000 7f175681fa40 -1 Falling back to public interface 2026-02-20T16:36:42.963 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:42.956+0000 7f175681fa40 -1 osd.3 41 log_to_monitors true 2026-02-20T16:36:43.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:43.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:43.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:43.242 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:43.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:43.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [38,41) [v2:127.0.0.1:6826/3007721000,v1:127.0.0.1:6827/3007721000] [v2:127.0.0.1:6828/3007721000,v1:127.0.0.1:6829/3007721000] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:43.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:43.404 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:43.404 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:43.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:43.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:43.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:43.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:43.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:43.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:43.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:36:43.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:36:43.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:36:43.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:43.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:43.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672971 2026-02-20T16:36:43.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672971 2026-02-20T16:36:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2026-02-20T16:36:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:43.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:43.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509450 2026-02-20T16:36:43.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509450 2026-02-20T16:36:43.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509450' 2026-02-20T16:36:43.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:43.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:36:43.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-20T16:36:43.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-20T16:36:43.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509450 3-193273528322' 2026-02-20T16:36:43.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:43.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:36:43.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:43.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:43.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:36:43.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:43.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:36:43.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:36:43.960 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836491 2026-02-20T16:36:43.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:44.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:36:44.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:45.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:45.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:45.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:36:45.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:45.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672971 2026-02-20T16:36:45.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:45.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:45.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672971 2026-02-20T16:36:45.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:45.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672971 2026-02-20T16:36:45.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2026-02-20T16:36:45.284 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672971 2026-02-20T16:36:45.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:45.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672971 -lt 42949672971 2026-02-20T16:36:45.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:45.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509450 2026-02-20T16:36:45.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:45.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:45.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509450 2026-02-20T16:36:45.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:45.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509450 2026-02-20T16:36:45.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2026-02-20T16:36:45.458 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509450 2026-02-20T16:36:45.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:45.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509450 2026-02-20T16:36:45.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:45.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-193273528322 2026-02-20T16:36:45.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:45.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:36:45.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-193273528322 2026-02-20T16:36:45.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:45.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-20T16:36:45.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 193273528322' 2026-02-20T16:36:45.633 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 193273528322 2026-02-20T16:36:45.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:36:45.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-20T16:36:45.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:45.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:45.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:45.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:45.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:45.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:46.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:36:46.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:46.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:46.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:46.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:46.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:46.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:46.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:36:46.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:36:46.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:36:46.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:36:46.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:36:46.869 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:36:46.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:46.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:46.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:46.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:46.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:46.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:46.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:46.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:46.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:36:46.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:36:46.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:36:46.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:36:46.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:36:46.995 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:46.988+0000 7fdbdbbf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:46.997 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:46.992+0000 7fdbdbbf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:46.998 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:46.992+0000 7fdbdbbf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:47.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:47.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:47.434 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:47.428+0000 7fdbdbbf1a40 -1 Falling back to public interface 2026-02-20T16:36:48.116 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:48.108+0000 7fdbdbbf1a40 -1 osd.1 46 log_to_monitors true 2026-02-20T16:36:48.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:36:48.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:48.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:36:48.298 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:48.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:48.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 47 last_clean_interval [10,46) [v2:127.0.0.1:6810/1200157628,v1:127.0.0.1:6811/1200157628] [v2:127.0.0.1:6812/1200157628,v1:127.0.0.1:6813/1200157628] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:36:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:36:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:36:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:36:48.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:36:48.455 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:36:48.455 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:36:48.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:36:48.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:36:48.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:36:48.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:36:48.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:36:48.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:36:48.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:36:48.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:36:48.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:36:48.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:36:48.721 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:36:48.721 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:36:48.722 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:36:48.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:36:48.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:48.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:36:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:36:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:36:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:36:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:48.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:36:48.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364802 2026-02-20T16:36:48.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364802 2026-02-20T16:36:48.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802' 2026-02-20T16:36:48.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:48.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:36:48.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509452 2026-02-20T16:36:48.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509452 2026-02-20T16:36:48.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-64424509452' 2026-02-20T16:36:48.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:36:48.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:36:49.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-20T16:36:49.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-20T16:36:49.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-64424509452 3-193273528324' 2026-02-20T16:36:49.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:49.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:36:49.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:49.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:36:49.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:36:49.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:49.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:36:49.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:36:49.016 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:36:49.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:36:49.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T16:36:49.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:49.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364802 2026-02-20T16:36:49.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:49.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:36:49.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364802 2026-02-20T16:36:49.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:49.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364802 2026-02-20T16:36:49.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364802' 2026-02-20T16:36:49.177 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 214748364802 2026-02-20T16:36:49.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:36:49.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364802 -lt 214748364802 2026-02-20T16:36:49.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:49.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509452 2026-02-20T16:36:49.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:49.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:36:49.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509452 2026-02-20T16:36:49.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:49.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509452 2026-02-20T16:36:49.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2026-02-20T16:36:49.341 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509452 2026-02-20T16:36:49.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:49.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509451 -lt 64424509452 2026-02-20T16:36:49.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:50.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:36:50.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:50.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509451 -lt 64424509452 2026-02-20T16:36:50.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:36:51.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:36:51.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:36:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509452 2026-02-20T16:36:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:36:51.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-193273528324 2026-02-20T16:36:51.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:36:51.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:36:51.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-193273528324 2026-02-20T16:36:51.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:36:51.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-20T16:36:51.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 193273528324' 2026-02-20T16:36:51.821 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 193273528324 2026-02-20T16:36:51.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:36:51.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528324 -lt 193273528324 2026-02-20T16:36:51.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:36:51.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:51.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:52.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:36:52.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:36:52.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:36:52.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:36:52.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:36:52.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:36:52.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:36:52.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:36:52.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:36:52.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:36:52.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:36:52.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:36:52.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:36:52.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:36:52.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:52.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:52.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:52.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:52.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:52.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:36:52.978 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:52.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:52.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:52.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:53.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:53.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:54.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:54.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:54.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:54.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:54.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:54.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:54.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:54.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:54.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:55.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:55.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:55.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:55.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:55.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:55.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:55.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:55.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:55.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:56.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:56.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:56.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:56.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:56.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:56.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:56.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:56.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:56.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:57.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:57.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:57.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:57.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:57.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:57.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:57.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:57.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:39.246577+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:57.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:36:58.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:36:58.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:36:58.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:36:58.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:36:58.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:36:58.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:36:58.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:39.246577+0000 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:36:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:36:58.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:36:58.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:36:58.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:36:59.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:36:59.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:36:59.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:36:59.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:36:59.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:36:59.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:36:59.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:59.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:36:59.311 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:36:59.311 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:36:59.312 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:36:59.312 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:36:59.312 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:36:59.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:36:59.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:36:59.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:36:59.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:36:59.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:36:59.434 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:36:59.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:36:59.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:36:59.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:36:59.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:36:59.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:36:59.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:36:59.449 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:59.444+0000 7f981f204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:59.450 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:59.444+0000 7f981f204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:59.451 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:59.444+0000 7f981f204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:36:59.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:36:59.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:36:59.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:36:59.878 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:36:59.872+0000 7f981f204a40 -1 Falling back to public interface 2026-02-20T16:37:00.475 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:00.468+0000 7f981f204a40 -1 osd.3 52 log_to_monitors true 2026-02-20T16:37:00.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:00.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:00.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:00.751 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:37:00.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:00.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:00.909 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 56 up_thru 56 down_at 53 last_clean_interval [45,52) [v2:127.0.0.1:6826/2233513067,v1:127.0.0.1:6827/2233513067] [v2:127.0.0.1:6828/2233513067,v1:127.0.0.1:6829/2233513067] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:37:00.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:00.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:00.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:00.911 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:00.911 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:37:00.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:00.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:00.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:01.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:01.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:01.184 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:37:01.184 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:37:01.184 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:37:01.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:01.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:01.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:01.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-20T16:37:01.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-20T16:37:01.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-20T16:37:01.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:01.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364805 2026-02-20T16:37:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364805 2026-02-20T16:37:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805' 2026-02-20T16:37:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:01.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:01.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-20T16:37:01.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-20T16:37:01.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-64424509455' 2026-02-20T16:37:01.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:01.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:01.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168578 2026-02-20T16:37:01.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168578 2026-02-20T16:37:01.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-64424509455 3-240518168578' 2026-02-20T16:37:01.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:01.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-20T16:37:01.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:01.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:01.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-20T16:37:01.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:01.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-20T16:37:01.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-20T16:37:01.484 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836497 2026-02-20T16:37:01.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:01.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-20T16:37:01.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:02.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:02.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:02.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-20T16:37:02.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:03.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:37:03.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:03.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-20T16:37:03.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:03.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364805 2026-02-20T16:37:03.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:03.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:03.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364805 2026-02-20T16:37:03.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:03.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364805 2026-02-20T16:37:03.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364805' 2026-02-20T16:37:03.977 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 214748364805 2026-02-20T16:37:03.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:04.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364805 -lt 214748364805 2026-02-20T16:37:04.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:04.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-20T16:37:04.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:04.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:04.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-20T16:37:04.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:04.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-20T16:37:04.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-20T16:37:04.139 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509455 2026-02-20T16:37:04.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:04.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509456 -lt 64424509455 2026-02-20T16:37:04.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:04.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168578 2026-02-20T16:37:04.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:04.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:04.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168578 2026-02-20T16:37:04.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:04.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168578 2026-02-20T16:37:04.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168578' 2026-02-20T16:37:04.314 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 240518168578 2026-02-20T16:37:04.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:04.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168578 -lt 240518168578 2026-02-20T16:37:04.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:04.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:04.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:04.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:04.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:04.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:04.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:04.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:05.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:05.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:05.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:05.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:37:05.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:37:05.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-20T16:37:05.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:37:05.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 132366"' 2026-02-20T16:37:05.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 132366' 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 132367"' 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 132367' 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 132366 132367' 2026-02-20T16:37:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 132366 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/132369: /' 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:37:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/132371: /' 2026-02-20T16:37:05.677 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.677 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:37:05.678 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:37:05.679 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:05.680 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:05.681 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:05.682 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: start osd.2 2026-02-20T16:37:05.683 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:05.745 INFO:tasks.workunit.client.0.trial013.stderr:1132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:37:05.746 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:05.747 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:05.748 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:05.749 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:05.750 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: start osd.1 2026-02-20T16:37:05.751 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:07.170 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2026-02-20T16:37:05.692+0000 7f330794ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2026-02-20T16:37:05.696+0000 7f330794ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2026-02-20T16:37:05.696+0000 7f330794ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 0 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2026-02-20T16:37:06.236+0000 7f330794ba40 -1 Falling back to public interface 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2026-02-20T16:37:06.920+0000 7f330794ba40 -1 osd.2 58 log_to_monitors true 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: 1 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: osd.2 up in weight 1 up_from 62 up_thru 47 down_at 58 last_clean_interval [15,58) [v2:127.0.0.1:6818/232691678,v1:127.0.0.1:6819/232691678] [v2:127.0.0.1:6820/232691678,v1:127.0.0.1:6821/232691678] exists,up fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:07.171 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:07.172 INFO:tasks.workunit.client.0.trial013.stderr:132371: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132371: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_t32369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2026-02-20T16:37:05.760+0000 7fb95bfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2026-02-20T16:37:05.764+0000 7fb95bfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2026-02-20T16:37:05.764+0000 7fb95bfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:07.217 INFO:tasks.workunit.client.0.trial013.stderr:132369: 0 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2026-02-20T16:37:06.224+0000 7fb95bfd6a40 -1 Falling back to public interface 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2026-02-20T16:37:06.864+0000 7fb95bfd6a40 -1 osd.1 58 log_to_monitors true 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: 1 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: osd.1 up in weight 1 up_from 61 up_thru 61 down_at 58 last_clean_interval [50,58) [v2:127.0.0.1:6810/3552450314,v1:127.0.0.1:6811/3552450314] [v2:127.0.0.1:6812/3552450314,v1:127.0.0.1:6813/3552450314] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:07.218 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:07.219 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:07.219 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:07.219 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:07.219 INFO:tasks.workunit.client.0.trial013.stderr:132369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gimeout_delays: echo true 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:07.730 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: 1 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: 2 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: 3' 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:07.731 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=261993005058 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 261993005058 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-261993005058' 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972354 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972354 2026-02-20T16:37:07.732 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-261993005058 2-266287972354' 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168580 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168580 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-261993005058 2-266287972354 3-240518168580' 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:07.733 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubunet_timeout_delays: echo true 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:07.762 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: 1 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: 2 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: 3' 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=261993005059 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 261993005059 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-261993005059' 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972355 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972355 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-261993005059 2-266287972355' 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168581 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168581 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-261993005059 2-266287972355 3-240518168581' 2026-02-20T16:37:07.763 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:07.764 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-20T16:37:09.379 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.379 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:09.379 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: waiting osd.0 seq 21474836499 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-261993005058 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.380 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-261993005058 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=261993005058 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 261993005058' 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: waiting osd.1 seq 261993005058 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 261993005059 -lt 261993005058 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-266287972354 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-266287972354 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972354 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 266287972354' 2026-02-20T16:37:09.381 INFO:tasks.workunit.client.0.trial013.stderr:132371: waiting osd.2 seq 266287972354 2026-02-20T16:37:09.382 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:09.382 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972355 -lt 266287972354 2026-02-20T16:37:09.382 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: waiting osd.0 seq 21474836500 2026-02-20T16:37:09.419 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836500 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-261993005059 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-261993005059 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=261993005059 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 261993005059' 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: waiting osd.1 seq 261993005059 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 261993005059 -lt 261993005059 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-266287972355 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:09.420 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-266287972355 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972355 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 266287972355' 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: waiting osd.2 seq 266287972355 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972355 -lt 266287972355 2026-02-20T16:37:09.421 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh7: flush_pg_stats: echo 3-240518168580 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168580 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168580 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168580' 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: waiting osd.3 seq 240518168580 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168581 -lt 240518168580 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:10.089 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:132371: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:10.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr::2267: flush_pg_stats: echo 3-240518168581 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168581 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168581 2026-02-20T16:37:10.148 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168581' 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: waiting osd.3 seq 240518168581 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168581 -lt 240518168581 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:10.149 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:132369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:10.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 132367 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:37:10.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:37:10.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:37:10.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:37:10.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:37:10.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:37:10.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:37:10.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:37:10.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:10.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:10.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:10.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:10.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:37:10.617 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:10.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:10.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:10.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:10.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:11.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:11.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:11.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:11.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:11.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:11.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:11.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:11.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:11.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:12.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:12.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:13.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:13.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:14.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:14.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:14.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:14.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:14.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:14.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:14.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:14.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:14.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:15.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:15.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:15.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:15.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:15.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:15.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:15.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:15.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:15.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:16.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:16.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:16.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:16.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:16.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:16.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:16.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:36:53.093817+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:16.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:17.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:17.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:17.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:17.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:17.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:17.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:17.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:17.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:11.456427+0000 '>' 2026-02-20T16:36:53.093817+0000 2026-02-20T16:37:17.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:37:17.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:37:17.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 135228"' 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 135228' 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:37:17.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 135230"' 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 135230' 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 135228 135230' 2026-02-20T16:37:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:37:17.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/135231: /' 2026-02-20T16:37:17.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:17.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 135228 2026-02-20T16:37:17.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:37:17.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/135234: /' 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:18.248 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: _ 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: hinfo_key 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: snapset 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.249 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: _ 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: hinfo_key 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: snapset 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:18.250 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:18.252 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: start osd.1 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:37:18.253 INFO:tasks.workunit.client.0.trial013.stderr:135234: start osd.2 2026-02-20T16:37:19.731 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:19.731 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:37:19.731 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:37:19.731 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2026-02-20T16:37:18.263+0000 7f2ce162ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2026-02-20T16:37:18.267+0000 7f2ce162ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2026-02-20T16:37:18.267+0000 7f2ce162ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:19.732 INFO:tasks.workunit.client.0.trial013.stderr:135234: 0 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2026-02-20T16:37:18.755+0000 7f2ce162ba40 -1 Falling back to public interface 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2026-02-20T16:37:19.399+0000 7f2ce162ba40 -1 osd.2 63 log_to_monitors true 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: 1 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: osd.2 up in weight 1 up_from 67 up_thru 47 down_at 64 last_clean_interval [62,63) [v2:127.0.0.1:6818/88053138,v1:127.0.0.1:6819/88053138] [v2:127.0.0.1:6820/88053138,v1:127.0.0.1:6821/88053138] exists,up fb47810d-e5c7-4020-88ad-2a2eac97d9f6 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:19.733 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:19.734 INFO:tasks.workunit.client.0.trial013.stderr:135234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593:very_ops 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2026-02-20T16:37:18.263+0000 7fbd31995a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2026-02-20T16:37:18.267+0000 7fbd31995a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2026-02-20T16:37:18.267+0000 7fbd31995a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:37:19.736 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: 0 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2026-02-20T16:37:18.739+0000 7fbd31995a40 -1 Falling back to public interface 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2026-02-20T16:37:19.391+0000 7fbd31995a40 -1 osd.1 63 log_to_monitors true 2026-02-20T16:37:19.737 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: 1 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [61,63) [v2:127.0.0.1:6810/830962687,v1:127.0.0.1:6811/830962687] [v2:127.0.0.1:6812/830962687,v1:127.0.0.1:6813/830962687] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:19.738 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:19.739 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:19.739 INFO:tasks.workunit.client.0.trial013.stderr:135231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:20.289 INFO:tasks.workunit.client.0.trial013.stderr:135231: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:20.290 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: 1 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: 2 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: 3' 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-20T16:37:20.291 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808835' 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808835 2-287762808835' 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168585 2026-02-20T16:37:20.292 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168585 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808835 2-287762808835 3-240518168585' 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135231: / get_timeout_delays: echo true 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:20.293 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: 1 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: 2 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: 3' 2026-02-20T16:37:20.294 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808834' 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:20.295 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808834 2-287762808834' 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168584 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168584 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808834 2-287762808834 3-240518168584' 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:20.296 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-20T16:37:21.954 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.954 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:21.954 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-20T16:37:21.954 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.954 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: waiting osd.0 seq 21474836503 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836503 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.955 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: waiting osd.1 seq 287762808834 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808834 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:21.956 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808834 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: waiting osd.2 seq 287762808834 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-20T16:37:21.957 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:21.961 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers./home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.961 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: waiting osd.0 seq 21474836504 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:21.962 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808835 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808835 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: waiting osd.1 seq 287762808835 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:21.963 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: waiting osd.2 seq 287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-20T16:37:21.964 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:22.747 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpsh:2267: flush_pg_stats: echo 3-240518168584 2026-02-20T16:37:22.747 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:22.747 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:22.747 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168584 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168584 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168584' 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: waiting osd.3 seq 240518168584 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168586 -lt 240518168584 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:22.748 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:22.749 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:22.797 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local exprers.sh:2267: flush_pg_stats: echo 3-240518168585 2026-02-20T16:37:22.797 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:22.797 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168585 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168585 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168585' 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: waiting osd.3 seq 240518168585 2026-02-20T16:37:22.798 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168586 -lt 240518168585 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:22.799 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:22.800 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-20T16:37:22.801 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:22.802 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:23.508 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local ession 2026-02-20T16:37:23.508 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:23.508 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:23.508 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:23.509 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:23.510 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:23.511 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:23.511 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:23.511 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:23.511 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:23.511 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:23.539 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_actexpression 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:23.540 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:23.541 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:23.542 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:24.633 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:37:24.634 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:24.635 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hive_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:24.636 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:37:24.637 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:24.638 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeelpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:26.164 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:26.165 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:135231: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:26.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:26.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 135230 2026-02-20T16:37:26.183 INFO:tasks.workunit.client.0.trial013.stderr:rs.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:37:26.184 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:26.185 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:135234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:26.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:26.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:37:26.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:37:26.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:37:26.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:37:26.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:37:26.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:37:26.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:37:26.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-20T16:37:26.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 138037"' 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 138037' 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:26.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 138038"' 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 138038' 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 138037 138038' 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 138037 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/138040: /' 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:26.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/138042: /' 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:37:26.821 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:26.822 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:26.823 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:26.825 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: start osd.1 2026-02-20T16:37:26.826 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:1138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:37:26.852 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:37:26.853 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:26.854 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:37:26.854 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:26.854 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:26.855 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:26.855 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:26.855 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:37:26.855 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:26.856 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:26.857 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:26.857 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:37:26.857 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:37:26.857 INFO:tasks.workunit.client.0.trial013.stderr:138040: start osd.3 2026-02-20T16:37:26.857 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:28.318 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:37:28.318 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:37:28.318 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:28.318 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:28.318 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2026-02-20T16:37:26.839+0000 7f4fadbc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2026-02-20T16:37:26.843+0000 7f4fadbc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2026-02-20T16:37:26.843+0000 7f4fadbc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: 0 2026-02-20T16:37:28.319 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2026-02-20T16:37:27.355+0000 7f4fadbc4a40 -1 Falling back to public interface 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2026-02-20T16:37:28.055+0000 7f4fadbc4a40 -1 osd.1 68 log_to_monitors true 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:28.320 INFO:tasks.workunit.client.0.trial013.stderr:138042: 1 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: osd.1 up in weight 1 up_from 72 up_thru 67 down_at 69 last_clean_interval [67,68) [v2:127.0.0.1:6810/3281911170,v1:127.0.0.1:6811/3281911170] [v2:127.0.0.1:6812/3281911170,v1:127.0.0.1:6813/3281911170] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:28.321 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:28.322 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:28.322 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:28.322 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:28.322 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:28.322 INFO:tasks.workunit.client.0.trial013.stderr:138042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:28.344 INFO:tasks.workunit.client.0.trial013.stderr:138042: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g38040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:37:28.344 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:37:28.344 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:28.344 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:28.344 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2026-02-20T16:37:26.867+0000 7f6a866a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2026-02-20T16:37:26.867+0000 7f6a866a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2026-02-20T16:37:26.867+0000 7f6a866a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: 0 2026-02-20T16:37:28.345 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2026-02-20T16:37:27.371+0000 7f6a866a3a40 -1 Falling back to public interface 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2026-02-20T16:37:28.099+0000 7f6a866a3a40 -1 osd.3 68 log_to_monitors true 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: 1 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [56,68) [v2:127.0.0.1:6826/422181186,v1:127.0.0.1:6827/422181186] [v2:127.0.0.1:6828/422181186,v1:127.0.0.1:6829/422181186] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:37:28.346 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:28.347 INFO:tasks.workunit.client.0.trial013.stderr:138040: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138040: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_tet_timeout_delays: echo true 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: 1 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: 2 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: 3' 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836507 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836507 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507' 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-20T16:37:28.885 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-309237645314' 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808837 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808837 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-309237645314 2-287762808837' 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-309237645314 2-287762808837 3-309237645314' 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:28.886 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836507 2026-02-20T16:37:28.892 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/imeout_delays: echo true 2026-02-20T16:37:28.892 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:28.892 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:28.892 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: 1 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: 2 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: 3' 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836508 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836508 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508' 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645315 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645315 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-309237645315' 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808838 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808838 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-309237645315 2-287762808838' 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645315 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645315 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-309237645315 2-287762808838 3-309237645315' 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:28.893 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836508 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836508 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836508 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: waiting osd.0 seq 21474836508 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836508 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-309237645315 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-309237645315 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645315 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: waiting osd.1 seq 309237645315 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645315 -lt 309237645315 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808838 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808838 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808838 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808838' 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: waiting osd.2 seq 287762808838 2026-02-20T16:37:29.382 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808838 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-309237645315 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:29.383 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-309237645315 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cuubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836507 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836507 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: waiting osd.0 seq 21474836507 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836507 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: waiting osd.1 seq 309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645315 -lt 309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808837 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808837 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808837 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808837' 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: waiting osd.2 seq 287762808837 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808837 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-309237645314 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:29.391 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-309237645314 2026-02-20T16:37:30.089 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats:t -d - -f 2 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645315 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 309237645315' 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: waiting osd.3 seq 309237645315 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645315 -lt 309237645315 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:30.090 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:30.091 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:30.092 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:30.092 INFO:tasks.workunit.client.0.trial013.stderr:138040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:30.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:30.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:30.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 138038 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr: cut -d - -f 2 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: waiting osd.3 seq 309237645314 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645315 -lt 309237645314 2026-02-20T16:37:30.119 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:30.120 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:138042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:30.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:30.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:37:30.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:30.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:30.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:30.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:37:30.586 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:37:30.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:30.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:30.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:30.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:30.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:11.456427+0000 '>' 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:30.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:31.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:31.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:31.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:31.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:31.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:11.456427+0000 '>' 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:31.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:32.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:32.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:32.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:32.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:32.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:32.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:32.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:33.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:11.456427+0000 '>' 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:33.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:34.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:34.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:34.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:34.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:34.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:34.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:34.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:11.456427+0000 '>' 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:37:35.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:37:35.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:37:35.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:37:35.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:37:35.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:37:35.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:37:35.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:37:31.143594+0000 '>' 2026-02-20T16:37:11.456427+0000 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 140808"' 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 140808' 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:37:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 140809"' 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 140809' 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 140808 140809' 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 140808 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/140811: /' 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/140813: /' 2026-02-20T16:37:35.900 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.900 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:35.901 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: _ 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: hinfo_key 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: snapset 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:35.902 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:37:35.903 INFO:tasks.workunit.client.0.trial013.stderr:140811: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: _ 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: hinfo_key 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: snapset 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:14s.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:37:35.904 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:35.905 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:37:35.906 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140813: start osd.1 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco0811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:37:35.907 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:35.908 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:37:35.909 INFO:tasks.workunit.client.0.trial013.stderr:140811: start osd.3 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2026-02-20T16:37:35.915+0000 7f589ae03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2026-02-20T16:37:35.919+0000 7f589ae03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2026-02-20T16:37:35.919+0000 7f589ae03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:37.393 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: 0 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2026-02-20T16:37:36.403+0000 7f589ae03a40 -1 Falling back to public interface 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: 1 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2026-02-20T16:37:37.243+0000 7f589ae03a40 -1 osd.3 74 log_to_monitors true 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: osd.3 up in weight 1 up_from 78 up_thru 72 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6826/2412727812,v1:127.0.0.1:6827/2412727812] [v2:127.0.0.1:6828/2412727812,v1:127.0.0.1:6829/2412727812] exists,up e8334c2d-4a4c-4b81-9b53-6101f9f010cb 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:37.394 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:37.395 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:37.399 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delvery_ops 2026-02-20T16:37:37.399 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:37:37.399 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:37:37.399 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2026-02-20T16:37:35.915+0000 7ff1936dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2026-02-20T16:37:35.915+0000 7ff1936dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2026-02-20T16:37:35.919+0000 7ff1936dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:37.400 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: 0 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2026-02-20T16:37:36.383+0000 7ff1936dca40 -1 Falling back to public interface 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2026-02-20T16:37:37.203+0000 7ff1936dca40 -1 osd.1 74 log_to_monitors true 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: 1 2026-02-20T16:37:37.401 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: osd.1 up in weight 1 up_from 78 up_thru 72 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6810/3812414857,v1:127.0.0.1:6811/3812414857] [v2:127.0.0.1:6812/3812414857,v1:127.0.0.1:6813/3812414857] exists,up 840f1ec4-9418-4c4f-af60-4abb65283728 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:37.402 INFO:tasks.workunit.client.0.trial013.stderr:140813: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:37.955 INFO:tasks.workunit.client.0.trial013.stderr:140813: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:37:37.955 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:37.955 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:37.955 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:37.955 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: 1 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: 2 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: 3' 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.956 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-335007449090' 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808841 2026-02-20T16:37:37.957 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808841 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-335007449090 2-287762808841' 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-335007449090 2-287762808841 3-335007449090' 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:37.958 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:14081ays 90 .1 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:37.959 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: 1 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: 2 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: 3' 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-20T16:37:37.960 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449091 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449091 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-335007449091' 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808842 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808842 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-335007449091 2-287762808842' 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449091 2026-02-20T16:37:37.961 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449091 2026-02-20T16:37:37.962 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-335007449091 2-287762808842 3-335007449091' 2026-02-20T16:37:39.612 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:39.612 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: waiting osd.0 seq 21474836512 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836512 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:39.613 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836512 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449091 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449091 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449091 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449091' 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: waiting osd.1 seq 335007449091 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:39.614 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449091 -lt 335007449091 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808842 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808842 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808842 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808842' 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: waiting osd.2 seq 287762808842 2026-02-20T16:37:39.615 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:39.623 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 23: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.623 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:39.623 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-20T16:37:39.623 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.623 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: waiting osd.0 seq 21474836511 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836511 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836511 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449090 2026-02-20T16:37:39.624 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449090 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: waiting osd.1 seq 335007449090 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449091 -lt 335007449090 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808841 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:37:39.625 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808841 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808841 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808841' 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: waiting osd.2 seq 287762808841 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808842 -lt 287762808841 2026-02-20T16:37:39.626 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-87762808842 -lt 287762808842 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-335007449091 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-335007449091 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449091 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: waiting osd.3 seq 335007449091 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:40.324 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449091 -lt 335007449091 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:140811: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:37:40.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 140809 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:helpers.sh:2267: flush_pg_stats: echo 3-335007449090 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-335007449090 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 335007449090' 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: waiting osd.3 seq 335007449090 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449091 -lt 335007449090 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:37:40.345 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:140813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:37:40.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:37:40.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:37:40.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:37:40.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:37:40.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:37:40.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:40.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:40.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:40.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:40.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:40.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:40.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:37:40.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:37:40.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:37:40.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:37:40.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:37:40.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:37:40.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:37:40.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:37:40.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:37:40.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:37:40.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:37:40.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:37:40.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:37:40.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:37:40.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:37:40.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:37:40.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:37:40.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:37:40.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:37:40.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:37:40.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:37:40.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:37:40.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:37:40.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:37:40.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:37:40.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:37:40.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:37:40.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:37:40.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:37:40.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:37:40.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:37:40.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:37:40.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:37:40.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:37:40.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:37:40.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:37:40.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:37:40.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:37:40.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:37:40.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:37:40.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:37:40.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:37:40.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:37:40.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=true 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-20T16:37:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:37:40.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:37:40.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:37:40.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:37:40.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:37:40.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:37:40.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:40.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:40.548 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:40.548 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.548 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:40.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:37:40.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:37:40.584 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.585 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:37:40.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:37:40.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:37:40.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:37:40.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:37:40.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:37:40.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:37:40.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:37:40.644 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:37:40.644 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:37:40.644 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:37:40.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:37:40.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-20T16:37:40.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:37:40.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:37:40.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:37:40.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:37:40.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:37:40.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:37:40.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:40.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:40.795 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:40.795 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.795 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:40.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:37:40.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:37:40.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-20T16:37:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:37:40.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:40.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:37:40.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:37:40.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:37:40.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dd1b3810-074b-413d-a5ab-3185cd79a864 2026-02-20T16:37:40.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dd1b3810-074b-413d-a5ab-3185cd79a864' 2026-02-20T16:37:40.823 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 dd1b3810-074b-413d-a5ab-3185cd79a864 2026-02-20T16:37:40.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:37:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUjZhpFkygMRAAFCgeNq2hu1Q+5VLZMVW5SQ== 2026-02-20T16:37:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUjZhpFkygMRAAFCgeNq2hu1Q+5VLZMVW5SQ=="}' 2026-02-20T16:37:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dd1b3810-074b-413d-a5ab-3185cd79a864 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:37:40.923 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:37:40.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:37:40.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUjZhpFkygMRAAFCgeNq2hu1Q+5VLZMVW5SQ== --osd-uuid dd1b3810-074b-413d-a5ab-3185cd79a864 2026-02-20T16:37:40.949 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:40.947+0000 7f4f17532a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:40.951 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:40.947+0000 7f4f17532a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:40.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:40.947+0000 7f4f17532a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:40.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:40.951+0000 7f4f17532a40 -1 bdev(0x56230f99ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:37:40.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:40.951+0000 7f4f17532a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:37:41.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:37:41.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:37:41.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:37:41.649 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:37:41.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:37:41.743 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:37:41.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:37:41.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:41.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:37:41.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:37:41.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:37:41.758 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:41.755+0000 7f5ae5682a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:41.760 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:41.755+0000 7f5ae5682a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:41.760 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:41.759+0000 7f5ae5682a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:41.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:41.839 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:37:41.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:41.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:37:41.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:42.174 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:42.171+0000 7f5ae5682a40 -1 Falling back to public interface 2026-02-20T16:37:42.785 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:42.783+0000 7f5ae5682a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:37:43.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:44.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:37:44.235 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2525677802,v1:127.0.0.1:6801/2525677802] [v2:127.0.0.1:6802/2525677802,v1:127.0.0.1:6803/2525677802] exists,up dd1b3810-074b-413d-a5ab-3185cd79a864 2026-02-20T16:37:44.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:44.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:44.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:44.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:44.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:37:44.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:37:44.245 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:37:44.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77' 2026-02-20T16:37:44.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:37:44.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYjZhpyGXnDhAAsU3jecEbAk7MLHR407d5ug== 2026-02-20T16:37:44.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYjZhpyGXnDhAAsU3jecEbAk7MLHR407d5ug=="}' 2026-02-20T16:37:44.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:37:44.416 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:37:44.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:37:44.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYjZhpyGXnDhAAsU3jecEbAk7MLHR407d5ug== --osd-uuid d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:37:44.443 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:44.439+0000 7f3b465fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:44.444 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:44.439+0000 7f3b465fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:44.445 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:44.443+0000 7f3b465fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:44.446 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:44.443+0000 7f3b465fda40 -1 bdev(0x55b80b792c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:37:44.446 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:44.443+0000 7f3b465fda40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:37:45.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:37:45.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:37:45.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:37:45.085 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:37:45.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:37:45.290 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:37:45.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:37:45.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:45.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:37:45.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:37:45.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:37:45.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:45.311+0000 7f5ec10eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:45.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:45.315+0000 7f5ec10eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:45.318 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:45.315+0000 7f5ec10eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:37:45.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:45.452 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:37:45.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:45.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:45.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:45.686 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:45.683+0000 7f5ec10eea40 -1 Falling back to public interface 2026-02-20T16:37:46.303 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:46.299+0000 7f5ec10eea40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:37:46.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:46.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:46.616 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:37:46.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:46.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:46.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:46.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:47.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:37:47.933 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2989000226,v1:127.0.0.1:6811/2989000226] [v2:127.0.0.1:6812/2989000226,v1:127.0.0.1:6813/2989000226] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:47.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:47.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:37:47.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:37:47.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:37:47.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:37:47.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e23bf3b1-ab41-473c-8557-ef84734e5660' 2026-02-20T16:37:47.938 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:37:47.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:37:47.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbjZhpvvqhOBAAr55CO8lwo79lrxrxKarrBg== 2026-02-20T16:37:47.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbjZhpvvqhOBAAr55CO8lwo79lrxrxKarrBg=="}' 2026-02-20T16:37:47.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e23bf3b1-ab41-473c-8557-ef84734e5660 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:37:48.119 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:37:48.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:37:48.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbjZhpvvqhOBAAr55CO8lwo79lrxrxKarrBg== --osd-uuid e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:37:48.146 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.143+0000 7f485121da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:48.148 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.143+0000 7f485121da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:48.149 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.147+0000 7f485121da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:48.149 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.147+0000 7f485121da40 -1 bdev(0x555bf5be6c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:37:48.149 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.147+0000 7f485121da40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:37:48.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:37:48.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:37:48.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:37:48.762 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:37:48.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:37:48.966 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:37:48.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:37:48.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:48.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:37:48.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:37:48.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:37:48.990 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.987+0000 7ffaefb9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:48.991 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.987+0000 7ffaefb9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:48.992 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:48.991+0000 7ffaefb9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:49.124 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:37:49.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:49.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:49.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:49.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:49.378 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:49.375+0000 7ffaefb9fa40 -1 Falling back to public interface 2026-02-20T16:37:50.059 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:50.055+0000 7ffaefb9fa40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:37:50.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:50.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:50.288 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:37:50.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:50.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:50.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:50.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:51.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:51.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:51.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:37:51.444 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:37:51.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:51.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:37:51.604 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2920789247,v1:127.0.0.1:6819/2920789247] [v2:127.0.0.1:6820/2920789247,v1:127.0.0.1:6821/2920789247] exists,up e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:37:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-20T16:37:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:37:51.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:37:51.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:37:51.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:37:51.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:37:51.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:37:51.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 330e3115-06f4-437c-9225-49ec0fdc513a' 2026-02-20T16:37:51.613 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:37:51.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:37:51.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDfjZhpIH0EJRAAztk1TCYiVR76lCUfwXU4DA== 2026-02-20T16:37:51.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDfjZhpIH0EJRAAztk1TCYiVR76lCUfwXU4DA=="}' 2026-02-20T16:37:51.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 330e3115-06f4-437c-9225-49ec0fdc513a -i td/osd-scrub-repair/3/new.json 2026-02-20T16:37:51.787 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T16:37:51.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T16:37:51.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfjZhpIH0EJRAAztk1TCYiVR76lCUfwXU4DA== --osd-uuid 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:37:51.821 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:51.819+0000 7f6d902d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:51.823 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:51.819+0000 7f6d902d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:51.824 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:51.819+0000 7f6d902d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:51.824 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:51.823+0000 7f6d902d3a40 -1 bdev(0x562f67456c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T16:37:51.825 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:51.823+0000 7f6d902d3a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T16:37:52.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T16:37:52.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:37:52.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T16:37:52.461 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T16:37:52.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:37:52.668 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T16:37:52.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T16:37:52.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:37:52.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:37:52.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:37:52.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:37:52.693 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:52.691+0000 7fcb6bcb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:52.694 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:52.691+0000 7fcb6bcb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:52.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:52.691+0000 7fcb6bcb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:37:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T16:37:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:37:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:37:52.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:37:52.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:37:52.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:52.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:37:52.833 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:37:52.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:52.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:52.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:53.082 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:53.079+0000 7fcb6bcb8a40 -1 Falling back to public interface 2026-02-20T16:37:53.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:37:53.647+0000 7fcb6bcb8a40 -1 osd.3 0 log_to_monitors true 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:53.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:54.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:37:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:37:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:37:55.150 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:37:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:37:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:37:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:37:55.311 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4184177032,v1:127.0.0.1:6827/4184177032] [v2:127.0.0.1:6828/4184177032,v1:127.0.0.1:6829/4184177032] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:37:55.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:37:55.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:37:55.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:37:55.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-20T16:37:55.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:37:55.463 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:37:55.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:37:55.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:37:55.700 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:37:55.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:37:56.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:37:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-20T16:37:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:37:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:37:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:37:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:37:57.014 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:37:57.014 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:37:57.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:37:57.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:37:57.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:37:57.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:37:57.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:37:57.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:37:57.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:37:57.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:37:57.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:37:57.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:37:57.279 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:37:57.279 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:37:57.279 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:37:57.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:37:57.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:57.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:37:57.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T16:37:57.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T16:37:57.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T16:37:57.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:57.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:37:57.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:37:57.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:37:57.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-20T16:37:57.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:57.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:37:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-20T16:37:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-20T16:37:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574851' 2026-02-20T16:37:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:37:57.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:37:57.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444035 2026-02-20T16:37:57.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444035 2026-02-20T16:37:57.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574851 3-73014444035' 2026-02-20T16:37:57.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:37:57.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T16:37:57.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:37:57.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:37:57.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T16:37:57.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:37:57.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T16:37:57.575 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T16:37:57.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T16:37:57.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:57.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:37:57.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:58.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:37:58.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:37:58.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:37:58.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:37:59.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:37:59.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:00.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T16:38:00.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:00.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:38:00.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:00.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:00.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:38:00.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:00.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:38:00.068 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:38:00.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:38:00.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:00.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705667 2026-02-20T16:38:00.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:00.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-20T16:38:00.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:00.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:00.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-20T16:38:00.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:00.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-20T16:38:00.242 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574851 2026-02-20T16:38:00.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-20T16:38:00.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:00.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-20T16:38:00.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:00.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444035 2026-02-20T16:38:00.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:00.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:00.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444035 2026-02-20T16:38:00.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:00.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444035 2026-02-20T16:38:00.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444035' 2026-02-20T16:38:00.413 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 73014444035 2026-02-20T16:38:00.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:00.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444035 -lt 73014444035 2026-02-20T16:38:00.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:00.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:00.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:00.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:00.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:00.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:38:00.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:00.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:00.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:38:01.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-20T16:38:01.338 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:38:01.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:38:01.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:38:01.643 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:38:01.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:38:02.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T16:38:02.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T16:38:02.855 INFO:tasks.workunit.client.0.trial013.stderr:set pool 2 allow_ec_overwrites to true 2026-02-20T16:38:02.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:38:02.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:02.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:02.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:02.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:02.868 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:02.868 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:02.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:02.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:02.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:02.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:02.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:02.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:02.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:02.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:02.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:03.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:03.147 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:03.148 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:03.148 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:38:03.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:03.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:03.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:03.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-20T16:38:03.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-20T16:38:03.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-20T16:38:03.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:03.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-20T16:38:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-20T16:38:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670' 2026-02-20T16:38:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:03.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:03.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-20T16:38:03.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-20T16:38:03.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-55834574853' 2026-02-20T16:38:03.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:03.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:03.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444037 2026-02-20T16:38:03.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444037 2026-02-20T16:38:03.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-55834574853 3-73014444037' 2026-02-20T16:38:03.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:03.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-20T16:38:03.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:03.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:03.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-20T16:38:03.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:03.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-20T16:38:03.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-20T16:38:03.447 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836486 2026-02-20T16:38:03.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:03.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-20T16:38:03.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:04.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:04.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:04.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-20T16:38:04.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:05.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:38:05.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:05.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836486 2026-02-20T16:38:05.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:05.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-20T16:38:05.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:05.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:05.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-20T16:38:05.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:05.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-20T16:38:05.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-20T16:38:05.941 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705670 2026-02-20T16:38:05.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:06.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-20T16:38:06.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:06.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574853 2026-02-20T16:38:06.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:06.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:06.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574853 2026-02-20T16:38:06.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574853 2026-02-20T16:38:06.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574853' 2026-02-20T16:38:06.107 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574853 2026-02-20T16:38:06.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:06.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-20T16:38:06.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:06.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444037 2026-02-20T16:38:06.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:06.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444037 2026-02-20T16:38:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:06.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444037 2026-02-20T16:38:06.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444037' 2026-02-20T16:38:06.283 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 73014444037 2026-02-20T16:38:06.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:06.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444038 -lt 73014444037 2026-02-20T16:38:06.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:06.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:06.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:06.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:06.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:06.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:06.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:06.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:06.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:06.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:06.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:06.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:38:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:38:06.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:38:07.232 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:38:07.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:38:07.432 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:38:07.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:38:07.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:38:07.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:38:07.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T16:38:07.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:38:07.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:38:07.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:38:07.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:38:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-20T16:38:07.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T16:38:07.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-20T16:38:07.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:38:07.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T16:38:07.629 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:38:07.629 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:38:07.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-20T16:38:07.790 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:07.790 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:07.790 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:38:07.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:07.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:07.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:07.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:07.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:08.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:08.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:38:08.296 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:08.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:08.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:08.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:08.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:08.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:08.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:08.410 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:08.406+0000 7f64bbffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:08.411 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:08.406+0000 7f64bbffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:08.412 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:08.410+0000 7f64bbffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:08.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:38:08.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:08.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:38:08.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:08.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:08.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:08.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:08.550 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:38:08.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:08.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:08.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:08.838 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:08.834+0000 7f64bbffda40 -1 Falling back to public interface 2026-02-20T16:38:09.480 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:09.478+0000 7f64bbffda40 -1 osd.3 34 log_to_monitors true 2026-02-20T16:38:09.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:09.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:09.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:09.713 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:09.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:09.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:09.868 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [17,34) [v2:127.0.0.1:6826/2398963882,v1:127.0.0.1:6827/2398963882] [v2:127.0.0.1:6828/2398963882,v1:127.0.0.1:6829/2398963882] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:38:09.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:09.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:09.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:09.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:09.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:09.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:09.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:09.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:09.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:09.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:09.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:09.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:09.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:09.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:09.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:09.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:09.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:09.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:09.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:10.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:10.137 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:10.137 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:10.137 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:38:10.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:10.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:10.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:10.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-20T16:38:10.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-20T16:38:10.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-20T16:38:10.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:10.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:10.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-20T16:38:10.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-20T16:38:10.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-20T16:38:10.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:10.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:10.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-20T16:38:10.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-20T16:38:10.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855' 2026-02-20T16:38:10.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:10.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:10.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-20T16:38:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-20T16:38:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-163208757250' 2026-02-20T16:38:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:10.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-20T16:38:10.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:10.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:10.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-20T16:38:10.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:10.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-20T16:38:10.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-20T16:38:10.432 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836489 2026-02-20T16:38:10.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-20T16:38:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:11.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:11.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:11.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-20T16:38:11.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:11.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-20T16:38:11.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:11.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:11.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-20T16:38:11.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:11.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-20T16:38:11.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-20T16:38:11.757 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705672 2026-02-20T16:38:11.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:11.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-20T16:38:11.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:11.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574855 2026-02-20T16:38:11.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:11.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:11.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574855 2026-02-20T16:38:11.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:11.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574855 2026-02-20T16:38:11.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2026-02-20T16:38:11.924 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574855 2026-02-20T16:38:11.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:12.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-20T16:38:12.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:12.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-163208757250 2026-02-20T16:38:12.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:12.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:12.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-163208757250 2026-02-20T16:38:12.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:12.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-20T16:38:12.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 163208757250' 2026-02-20T16:38:12.088 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 163208757250 2026-02-20T16:38:12.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:12.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-20T16:38:12.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:12.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:12.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:12.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:12.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:12.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:12.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:12.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:12.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:12.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:12.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:12.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:12.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:12.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:12.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:12.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:12.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:12.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:12.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:38:12.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:38:12.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:38:12.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:38:12.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:38:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:38:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:38:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:38:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:38:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:12.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:13.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:13.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:38:13.290 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:13.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:13.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:13.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:13.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:13.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:01.642461+0000 '>' 2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:13.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:14.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:14.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:14.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:14.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:14.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:14.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:14.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:14.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:01.642461+0000 '>' 2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:14.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:15.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:15.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:15.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:01.642461+0000 2026-02-20T16:38:15.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:38:15.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:38:15.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:15.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:15.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:15.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:15.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:16.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:16.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:16.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:38:16.172 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:38:16.172 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:38:16.172 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:38:16.173 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:38:16.173 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:38:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:38:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:38:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:16.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:16.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:16.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:16.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:16.305 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:16.302+0000 7f5640b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:16.307 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:16.302+0000 7f5640b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:16.307 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:16.302+0000 7f5640b0ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:16.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:38:16.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:16.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:16.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:16.726 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:16.722+0000 7f5640b0ba40 -1 Falling back to public interface 2026-02-20T16:38:17.307 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:17.302+0000 7f5640b0ba40 -1 osd.3 41 log_to_monitors true 2026-02-20T16:38:17.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:17.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:17.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:17.614 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:17.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:17.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:17.773 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [38,41) [v2:127.0.0.1:6826/4171166995,v1:127.0.0.1:6827/4171166995] [v2:127.0.0.1:6828/4171166995,v1:127.0.0.1:6829/4171166995] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:17.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:17.775 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:17.775 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:17.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:17.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:17.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:17.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:17.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:17.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:17.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:17.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:17.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:18.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:18.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:18.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:38:18.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:38:18.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:38:18.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:18.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:18.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705675 2026-02-20T16:38:18.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705675 2026-02-20T16:38:18.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675' 2026-02-20T16:38:18.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:18.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:18.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-20T16:38:18.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-20T16:38:18.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-55834574858' 2026-02-20T16:38:18.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:18.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:18.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-20T16:38:18.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-20T16:38:18.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-55834574858 3-193273528322' 2026-02-20T16:38:18.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:18.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:38:18.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:18.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:18.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:38:18.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:18.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:38:18.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:38:18.338 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836491 2026-02-20T16:38:18.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:18.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:38:18.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:19.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:19.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:19.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:38:19.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:19.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705675 2026-02-20T16:38:19.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:19.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:19.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705675 2026-02-20T16:38:19.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:19.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705675 2026-02-20T16:38:19.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-20T16:38:19.673 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705675 2026-02-20T16:38:19.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:19.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-20T16:38:19.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:19.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574858 2026-02-20T16:38:19.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:19.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:19.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574858 2026-02-20T16:38:19.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:19.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574858 2026-02-20T16:38:19.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2026-02-20T16:38:19.834 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574858 2026-02-20T16:38:19.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:19.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-20T16:38:19.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:19.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-193273528322 2026-02-20T16:38:19.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:20.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:20.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-193273528322 2026-02-20T16:38:20.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:20.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-20T16:38:20.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 193273528322' 2026-02-20T16:38:20.003 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 193273528322 2026-02-20T16:38:20.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:20.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-20T16:38:20.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:20.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:20.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:20.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:20.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:20.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:20.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:20.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:20.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:20.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:38:20.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:20.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:20.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:21.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:38:21.204 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:21.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:38:21.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:21.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:21.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:21.314 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:21.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:21.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:38:21.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:38:21.318 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:21.332 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:21.326+0000 7f80e226fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:21.333 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:21.330+0000 7f80e226fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:21.334 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:21.330+0000 7f80e226fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:21.474 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:38:21.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:21.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:21.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:21.762 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:21.758+0000 7f80e226fa40 -1 Falling back to public interface 2026-02-20T16:38:22.325 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:22.322+0000 7f80e226fa40 -1 osd.1 46 log_to_monitors true 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:22.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 47 last_clean_interval [9,46) [v2:127.0.0.1:6810/2104989604,v1:127.0.0.1:6811/2104989604] [v2:127.0.0.1:6812/2104989604,v1:127.0.0.1:6813/2104989604] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:22.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:22.803 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:22.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:22.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:22.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:22.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:22.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:22.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:22.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:22.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:22.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:22.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:23.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:23.076 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:23.076 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:23.077 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:38:23.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:23.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:23.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:23.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:38:23.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:38:23.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:38:23.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:23.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:23.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364802 2026-02-20T16:38:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364802 2026-02-20T16:38:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802' 2026-02-20T16:38:23.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:23.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:23.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-20T16:38:23.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-20T16:38:23.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-55834574860' 2026-02-20T16:38:23.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:23.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:23.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-20T16:38:23.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-20T16:38:23.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-55834574860 3-193273528324' 2026-02-20T16:38:23.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:23.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:38:23.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:23.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:23.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:38:23.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:23.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:38:23.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:38:23.367 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:38:23.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:23.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-20T16:38:23.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:24.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:24.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:24.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-20T16:38:24.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:25.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:38:25.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:25.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-20T16:38:25.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:25.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364802 2026-02-20T16:38:25.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:25.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:25.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364802 2026-02-20T16:38:25.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:25.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364802 2026-02-20T16:38:25.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364802' 2026-02-20T16:38:25.867 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 214748364802 2026-02-20T16:38:25.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:26.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364802 -lt 214748364802 2026-02-20T16:38:26.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:26.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574860 2026-02-20T16:38:26.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:26.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:26.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574860 2026-02-20T16:38:26.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:26.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574860 2026-02-20T16:38:26.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-20T16:38:26.044 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574860 2026-02-20T16:38:26.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:26.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-20T16:38:26.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:26.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-193273528324 2026-02-20T16:38:26.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:26.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:26.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-193273528324 2026-02-20T16:38:26.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:26.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-20T16:38:26.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 193273528324' 2026-02-20T16:38:26.216 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 193273528324 2026-02-20T16:38:26.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:26.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528324 -lt 193273528324 2026-02-20T16:38:26.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:26.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:26.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:26.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:26.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:26.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:26.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:26.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:26.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:26.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:26.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:26.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:26.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:26.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:26.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:38:26.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:38:26.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:38:26.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:27.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:27.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:27.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:27.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:27.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:38:27.399 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:27.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:27.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:27.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:28.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:28.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:28.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:28.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:28.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:28.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:28.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:29.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:29.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:29.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:29.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:29.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:29.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:29.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:29.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:29.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:30.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:30.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:30.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:30.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:30.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:30.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:30.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:31.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:31.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:32.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:32.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:32.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:32.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:32.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:32.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:32.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:32.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:13.527033+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:32.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:33.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:33.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:33.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:33.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:33.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:33.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:33.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:13.527033+0000 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:33.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:33.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:33.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:38:33.754 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:38:33.755 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:38:33.755 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:38:33.755 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:38:33.755 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:33.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:33.872 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:33.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:33.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:33.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:33.887 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:33.882+0000 7f0f6448ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:33.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:33.886+0000 7f0f6448ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:33.890 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:33.886+0000 7f0f6448ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:34.035 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:38:34.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:34.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:34.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:34.326 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:34.322+0000 7f0f6448ea40 -1 Falling back to public interface 2026-02-20T16:38:34.969 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:38:34.966+0000 7f0f6448ea40 -1 osd.3 53 log_to_monitors true 2026-02-20T16:38:35.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:35.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:35.190 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:35.343 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 57 up_thru 57 down_at 54 last_clean_interval [45,53) [v2:127.0.0.1:6826/896675713,v1:127.0.0.1:6827/896675713] [v2:127.0.0.1:6828/896675713,v1:127.0.0.1:6829/896675713] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:35.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:35.345 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:35.345 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:35.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:35.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:35.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:35.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:35.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:35.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:35.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-20T16:38:35.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-20T16:38:35.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-20T16:38:35.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:35.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:35.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364805 2026-02-20T16:38:35.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364805 2026-02-20T16:38:35.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805' 2026-02-20T16:38:35.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:35.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:35.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-20T16:38:35.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-20T16:38:35.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-55834574863' 2026-02-20T16:38:35.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:35.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:35.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=244813135874 2026-02-20T16:38:35.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 244813135874 2026-02-20T16:38:35.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-55834574863 3-244813135874' 2026-02-20T16:38:35.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:35.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-20T16:38:35.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:35.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:35.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-20T16:38:35.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:35.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-20T16:38:35.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-20T16:38:35.903 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836497 2026-02-20T16:38:35.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:36.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-20T16:38:36.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:37.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:37.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:37.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-20T16:38:37.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:37.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364805 2026-02-20T16:38:37.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:37.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:37.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364805 2026-02-20T16:38:37.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364805 2026-02-20T16:38:37.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364805' 2026-02-20T16:38:37.223 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 214748364805 2026-02-20T16:38:37.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:37.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364805 -lt 214748364805 2026-02-20T16:38:37.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:37.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-20T16:38:37.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:37.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:37.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574863 2026-02-20T16:38:37.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:37.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574863 2026-02-20T16:38:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-20T16:38:37.394 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574863 2026-02-20T16:38:37.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:37.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574864 -lt 55834574863 2026-02-20T16:38:37.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:37.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-244813135874 2026-02-20T16:38:37.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:37.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:37.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-244813135874 2026-02-20T16:38:37.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:37.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=244813135874 2026-02-20T16:38:37.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 244813135874' 2026-02-20T16:38:37.570 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 244813135874 2026-02-20T16:38:37.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:37.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 244813135874 -lt 244813135874 2026-02-20T16:38:37.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:37.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:37.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:37.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:37.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:37.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:38.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:38.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:38.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:38.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:38.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:38.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:38.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:38.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:38:38.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-20T16:38:38.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 155592"' 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 155592' 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 155594"' 2026-02-20T16:38:38.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 155594' 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 155592 155594' 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 155592 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/155595: /' 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:38:38.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/155598: /' 2026-02-20T16:38:38.948 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:38:38.949 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:38:38.950 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:38.951 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:38.952 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:38.952 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:38.952 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:38.953 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:38:38.954 INFO:tasks.workunit.client.0.trial013.stderr:155598: start osd.2 2026-02-20T16:38:38.955 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:1155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.976 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:38.977 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:38:38.978 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:38.979 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:38.980 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: start osd.1 2026-02-20T16:38:38.981 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2026-02-20T16:38:38.966+0000 7ff8793e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2026-02-20T16:38:38.970+0000 7ff8793e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2026-02-20T16:38:38.970+0000 7ff8793e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 0 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2026-02-20T16:38:39.486+0000 7ff8793e4a40 -1 Falling back to public interface 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 1 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2026-02-20T16:38:40.418+0000 7ff8793e4a40 -1 osd.2 59 log_to_monitors true 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:41.607 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: osd.2 up in weight 1 up_from 63 up_thru 47 down_at 60 last_clean_interval [13,59) [v2:127.0.0.1:6810/715322560,v1:127.0.0.1:6811/715322560] [v2:127.0.0.1:6812/715322560,v1:127.0.0.1:6813/715322560] exists,up e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:41.608 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.clie55595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2026-02-20T16:38:38.990+0000 7fc5e241aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2026-02-20T16:38:38.990+0000 7fc5e241aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2026-02-20T16:38:38.990+0000 7fc5e241aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:41.612 INFO:tasks.workunit.client.0.trial013.stderr:155595: 0 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2026-02-20T16:38:39.498+0000 7fc5e241aa40 -1 Falling back to public interface 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: 1 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2026-02-20T16:38:40.434+0000 7fc5e241aa40 -1 osd.1 60 log_to_monitors true 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: osd.1 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [50,60) [v2:127.0.0.1:6818/3740135253,v1:127.0.0.1:6819/3740135253] [v2:127.0.0.1:6820/3740135253,v1:127.0.0.1:6821/3740135253] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:41.613 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:42.101 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:42.101 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:42.101 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:42.101 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:42.101 INFO:tasks.workunit.client.0.trial013.stderr:155595: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:42.102 INFO:tasks.workunit.client.0.trial013.stderr:155595: 1 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: 2 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: 3' 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939651 2026-02-20T16:38:42.103 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939651 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-270582939651' 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939651 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939651 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-270582939651 2-270582939651' 2026-02-20T16:38:42.104 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stnt.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:42.164 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: 1 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: 2 2026-02-20T16:38:42.165 INFO:tasks.workunit.client.0.trial013.stderr:155598: 3' 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939650 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939650 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-270582939650' 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.166 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:42.167 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939650 2026-02-20T16:38:42.167 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939650 2026-02-20T16:38:42.167 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-270582939650 2-270582939650' 2026-02-20T16:38:42.167 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:42.167 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:43.661 INFO:tasks.workunit.client.0.trial013.stderr:ats 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=244813135877 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 244813135877 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-270582939651 2-270582939651 3-244813135877' 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:43.662 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: waiting osd.0 seq 21474836500 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836500 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-20T16:38:43.663 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-270582939651 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-270582939651 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=270582939651 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 270582939651' 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: waiting osd.1 seq 270582939651 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939651 -lt 270582939651 2026-02-20T16:38:43.664 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-270582939651 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-270582939651 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:43.665 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=244813135876 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 244813135876 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-270582939650 2-270582939650 3-244813135876' 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-20T16:38:43.666 INFO:tasks.workunit.client.0.trial013.stderr:155598: waiting osd.0 seq 21474836499 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-270582939650 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:43.667 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-270582939650 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=270582939650 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: waiting osd.1 seq 270582939650 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939651 -lt 270582939650 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-270582939650 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:43.668 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-270582939650 2026-02-20T16:38:43.669 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:44.527 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg__pg_stats: seq=270582939651 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 270582939651' 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: waiting osd.2 seq 270582939651 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939651 -lt 270582939651 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-244813135877 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:44.528 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-244813135877 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=244813135877 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 244813135877' 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: waiting osd.3 seq 244813135877 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 244813135877 -lt 244813135877 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:44.529 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:155595: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:44.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:38:44.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:38:44.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 155594 2026-02-20T16:38:44.558 INFO:tasks.workunit.client.0.trial013.stderr:stats: seq=270582939650 2026-02-20T16:38:44.558 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 270582939650' 2026-02-20T16:38:44.558 INFO:tasks.workunit.client.0.trial013.stderr:155598: waiting osd.2 seq 270582939650 2026-02-20T16:38:44.558 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939651 -lt 270582939650 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-244813135876 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-244813135876 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=244813135876 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 244813135876' 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: waiting osd.3 seq 244813135876 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:44.559 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 244813135877 -lt 244813135876 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:44.560 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:155598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:155598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:38:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:38:44.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:38:44.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:38:44.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:38:44.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:38:44.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:38:44.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:38:44.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:38:44.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:38:44.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:38:44.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:44.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:44.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:44.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:44.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:44.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:38:45.033 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:38:45.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:45.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:45.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:45.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:45.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:45.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:45.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:45.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:46.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:46.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:46.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:46.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:46.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:46.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:46.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:46.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:46.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:47.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:47.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:47.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:47.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:47.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:47.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:47.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:47.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:47.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:48.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:48.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:48.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:48.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:48.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:48.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:48.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:48.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:48.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:49.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:49.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:49.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:49.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:49.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:49.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:49.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:49.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:49.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:50.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:50.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:50.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:50.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:50.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:50.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:50.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:51.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:28.317850+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:51.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:38:52.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:38:52.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:38:52.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:38:52.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:38:52.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:38:52.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:38:52.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:38:52.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:28.317850+0000 2026-02-20T16:38:52.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:38:52.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:38:52.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 158502"' 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 158502' 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 158503"' 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 158503' 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:38:52.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 158502 158503' 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 158502 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/158505: /' 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:38:52.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/158507: /' 2026-02-20T16:38:52.660 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:38:52.661 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.662 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: _ 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: hinfo_key 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: snapset 2026-02-20T16:38:52.663 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: _ 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: hinfo_key 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: snapset 2026-02-20T16:38:52.664 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:52.665 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: start osd.1 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:38:52.666 INFO:tasks.workunit.client.0.trial013.stderr:158507: start osd.2 2026-02-20T16:38:54.147 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:54.147 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:38:54.147 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2026-02-20T16:38:52.674+0000 7fef8b9bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2026-02-20T16:38:52.678+0000 7fef8b9bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2026-02-20T16:38:52.678+0000 7fef8b9bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:54.148 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: 0 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2026-02-20T16:38:53.146+0000 7fef8b9bca40 -1 Falling back to public interface 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2026-02-20T16:38:53.778+0000 7fef8b9bca40 -1 osd.1 65 log_to_monitors true 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: 1 2026-02-20T16:38:54.149 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: osd.1 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [63,65) [v2:127.0.0.1:6818/175079882,v1:127.0.0.1:6819/175079882] [v2:127.0.0.1:6820/175079882,v1:127.0.0.1:6821/175079882] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:54.150 INFO:tasks.workunit.client.0.trial013.stderr:158505: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1very_ops 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2026-02-20T16:38:52.674+0000 7f47ef7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2026-02-20T16:38:52.678+0000 7f47ef7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2026-02-20T16:38:52.678+0000 7f47ef7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:54.151 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: 0 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2026-02-20T16:38:53.130+0000 7f47ef7bea40 -1 Falling back to public interface 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2026-02-20T16:38:53.766+0000 7f47ef7bea40 -1 osd.2 65 log_to_monitors true 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:54.152 INFO:tasks.workunit.client.0.trial013.stderr:158507: 1 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: osd.2 up in weight 1 up_from 69 up_thru 47 down_at 66 last_clean_interval [63,65) [v2:127.0.0.1:6810/2186114450,v1:127.0.0.1:6811/2186114450] [v2:127.0.0.1:6812/2186114450,v1:127.0.0.1:6813/2186114450] exists,up e23bf3b1-ab41-473c-8557-ef84734e5660 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:54.153 INFO:tasks.workunit.client.0.trial013.stderr:158507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: 1 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: 2 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: 3' 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743426' 2026-02-20T16:38:54.697 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743426 2-296352743426' 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=244813135880 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 244813135880 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743426 2-296352743426 3-244813135880' 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:54.698 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:15850593: get_timeout_delays: echo true 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: 1 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: 2 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: 3' 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-296352743427' 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-20T16:38:54.709 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-296352743427 2-296352743427' 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=244813135881 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 244813135881 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-296352743427 2-296352743427 3-244813135881' 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:54.710 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: waiting osd.0 seq 21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836505 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-296352743427 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:56.349 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 296352743427' 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: waiting osd.1 seq 296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: waiting osd.2 seq 296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-20T16:38:56.350 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:56.352 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help7: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: waiting osd.0 seq 21474836504 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 296352743426' 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: waiting osd.1 seq 296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: waiting osd.2 seq 296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743426 2026-02-20T16:38:56.353 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-ers.sh:2267: flush_pg_stats: echo 3-244813135881 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-244813135881 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=244813135881 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 244813135881' 2026-02-20T16:38:57.046 INFO:tasks.workunit.client.0.trial013.stderr:158505: waiting osd.3 seq 244813135881 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 244813135882 -lt 244813135881 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:57.047 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:158505: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:57.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:38:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:38:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 158503 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:helpers.sh:2267: flush_pg_stats: echo 3-244813135880 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-244813135880 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=244813135880 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 244813135880' 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: waiting osd.3 seq 244813135880 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:38:57.072 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 244813135882 -lt 244813135880 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:38:57.073 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:38:57.074 INFO:tasks.workunit.client.0.trial013.stderr:158507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:38:57.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:38:57.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:38:57.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:38:57.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:38:57.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:38:57.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:38:57.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:38:57.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:38:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-20T16:38:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 160775"' 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 160775' 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:38:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 160777"' 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 160777' 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 160775 160777' 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 160775 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/160778: /' 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/160781: /' 2026-02-20T16:38:57.762 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.762 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:57.762 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:38:57.762 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:57.762 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:57.763 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.764 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:57.765 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:38:57.766 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:38:57.767 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:38:57.768 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:38:57.769 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:57.769 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:57.769 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:38:57.769 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:38:57.769 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:57.770 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: start osd.3 2026-02-20T16:38:57.771 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:1activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:38:57.772 INFO:tasks.workunit.client.0.trial013.stderr:160781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:38:57.773 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:38:57.774 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:38:57.774 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:38:57.774 INFO:tasks.workunit.client.0.trial013.stderr:160781: start osd.1 2026-02-20T16:38:57.774 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:38:59.251 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2026-02-20T16:38:57.778+0000 7fb343920a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2026-02-20T16:38:57.778+0000 7fb343920a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2026-02-20T16:38:57.778+0000 7fb343920a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: 0 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:59.252 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2026-02-20T16:38:58.242+0000 7fb343920a40 -1 Falling back to public interface 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2026-02-20T16:38:58.890+0000 7fb343920a40 -1 osd.3 70 log_to_monitors true 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: 1 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: osd.3 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [57,70) [v2:127.0.0.1:6819/2660200820,v1:127.0.0.1:6821/2660200820] [v2:127.0.0.1:6823/2660200820,v1:127.0.0.1:6825/2660200820] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:59.253 INFO:tasks.workunit.client.0.trial013.stderr:160778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160778: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g60781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2026-02-20T16:38:57.778+0000 7f5b52197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2026-02-20T16:38:57.778+0000 7f5b52197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2026-02-20T16:38:57.778+0000 7f5b52197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: 0 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:38:59.254 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2026-02-20T16:38:58.242+0000 7f5b52197a40 -1 Falling back to public interface 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2026-02-20T16:38:58.902+0000 7f5b52197a40 -1 osd.1 70 log_to_monitors true 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: 1 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: osd.1 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [69,70) [v2:127.0.0.1:6818/3533701067,v1:127.0.0.1:6820/3533701067] [v2:127.0.0.1:6822/3533701067,v1:127.0.0.1:6824/3533701067] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:38:59.255 INFO:tasks.workunit.client.0.trial013.stderr:160781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:59.801 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: 1 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: 2 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: 3' 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836507 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836507 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507' 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547202 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547202 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-322122547202' 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:59.802 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743429 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743429 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-322122547202 2-296352743429' 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547202 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547202 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-322122547202 2-296352743429 3-322122547202' 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:59.803 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836507 2026-02-20T16:38:59.812 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/et_timeout_delays: echo true 2026-02-20T16:38:59.812 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:38:59.812 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:38:59.812 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: 1 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: 2 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: 3' 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:38:59.813 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836508 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836508 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508' 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547203 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547203 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-322122547203' 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-20T16:38:59.814 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-322122547203 2-296352743430' 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547203 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547203 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-322122547203 2-296352743430 3-322122547203' 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:38:59.815 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836508 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: waiting osd.0 seq 21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836507 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-322122547202 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.479 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-322122547202 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547202 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: waiting osd.1 seq 322122547202 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547202 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743429 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743429 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743429 2026-02-20T16:39:02.480 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743429' 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160778:ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836508 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836508 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-20T16:39:02.482 INFO:tasks.workunit.client.0.trial013.stderr:160781: waiting osd.0 seq 21474836508 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836508 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836508 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836508 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-322122547203 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-322122547203 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547203 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: waiting osd.1 seq 322122547203 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547203 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743430 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743430 2026-02-20T16:39:02.483 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160781: waiting osd.2 seq 296352743429 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743430 -lt 296352743429 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-322122547202 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:03.361 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-322122547202 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547202 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 322122547202' 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: waiting osd.3 seq 322122547202 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547202 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:39:03.362 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:39:03.363 INFO:tasks.workunit.client.0.trial013.stderr:160778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:39:03.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:39:03.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:39:03.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 160777 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr: waiting osd.2 seq 296352743430 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-322122547203 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:39:03.376 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-322122547203 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547203 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 322122547203' 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: waiting osd.3 seq 322122547203 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547203 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:39:03.377 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:03.378 INFO:tasks.workunit.client.0.trial013.stderr:160781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:03.379 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:39:03.379 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:39:03.379 INFO:tasks.workunit.client.0.trial013.stderr:160781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:39:03.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:39:03.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:39:03.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:39:03.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:39:03.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:39:03.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:39:03.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:39:03.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:39:03.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:39:03.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:03.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:03.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:03.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:03.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:03.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:39:03.848 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:39:03.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:04.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:39:05.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:39:05.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:05.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:05.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:05.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:05.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:05.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:05.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:05.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:39:06.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:39:06.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:06.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:06.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:06.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:06.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:06.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:06.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:06.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:39:07.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:39:07.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:07.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:07.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:07.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:07.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:07.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:07.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:07.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:39:08.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:39:08.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:08.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:08.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:08.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:08.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:08.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:08.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:38:46.066309+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:08.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:39:09.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:39:09.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:39:09.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:39:09.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:39:09.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:39:09.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:39:09.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:04.850639+0000 '>' 2026-02-20T16:38:46.066309+0000 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:39:09.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 163658"' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 163658' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 163660"' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 163660' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 163658 163660' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 163658 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/163661: /' 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:39:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/163664: /' 2026-02-20T16:39:10.344 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.344 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:39:10.344 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:39:10.344 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:39:10.345 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: _ 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: hinfo_key 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: snapset 2026-02-20T16:39:10.346 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:10.347 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.348 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:39:10.349 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: _ 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: hinfo_key 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: snapset 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:39:10.350 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:16s.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:39:10.351 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:10.352 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: start osd.1 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco3661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:10.353 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:10.354 INFO:tasks.workunit.client.0.trial013.stderr:163661: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:10.355 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:39:10.356 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:39:10.356 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:39:10.356 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:39:10.356 INFO:tasks.workunit.client.0.trial013.stderr:163661: start osd.3 2026-02-20T16:39:11.831 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2026-02-20T16:39:10.357+0000 7fb3066a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2026-02-20T16:39:10.361+0000 7fb3066a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2026-02-20T16:39:10.361+0000 7fb3066a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:39:11.832 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: 0 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:11.833 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2026-02-20T16:39:10.873+0000 7fb3066a5a40 -1 Falling back to public interface 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2026-02-20T16:39:11.521+0000 7fb3066a5a40 -1 osd.3 76 log_to_monitors true 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: 1 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: osd.3 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [75,76) [v2:127.0.0.1:6826/2853525219,v1:127.0.0.1:6827/2853525219] [v2:127.0.0.1:6828/2853525219,v1:127.0.0.1:6829/2853525219] exists,up 330e3115-06f4-437c-9225-49ec0fdc513a 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:11.834 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:39:11.835 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:39:11.835 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:39:11.835 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:39:11.842 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delvery_ops 2026-02-20T16:39:11.842 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:39:11.842 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2026-02-20T16:39:10.357+0000 7f4dacc89a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2026-02-20T16:39:10.361+0000 7f4dacc89a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2026-02-20T16:39:10.361+0000 7f4dacc89a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:11.843 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: 0 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2026-02-20T16:39:10.857+0000 7f4dacc89a40 -1 Falling back to public interface 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2026-02-20T16:39:11.521+0000 7f4dacc89a40 -1 osd.1 76 log_to_monitors true 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: 1 2026-02-20T16:39:11.844 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [75,76) [v2:127.0.0.1:6818/1417545873,v1:127.0.0.1:6819/1417545873] [v2:127.0.0.1:6820/1417545873,v1:127.0.0.1:6821/1417545873] exists,up d882b4f0-0f9a-4dd3-b707-32d8ddb3ce77 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:39:11.845 INFO:tasks.workunit.client.0.trial013.stderr:163664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: 1 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: 2 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: 3' 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383682' 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743433 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743433 2026-02-20T16:39:12.394 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383682 2-296352743433' 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383682 2-296352743433 3-343597383682' 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:16366ays 90 .1 2026-02-20T16:39:12.395 INFO:tasks.workunit.client.0.trial013.stderr:163661: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: 1 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: 2 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: 3' 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383683' 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383683 2-296352743434' 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-20T16:39:12.396 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383683 2-296352743434 3-343597383683' 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest4: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-20T16:39:14.048 INFO:tasks.workunit.client.0.trial013.stderr:163664: waiting osd.0 seq 21474836512 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836512 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836512 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383682 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383682 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: waiting osd.1 seq 343597383682 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743433 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743433 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743433 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743433' 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: waiting osd.2 seq 296352743433 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743434 -lt 296352743433 2026-02-20T16:39:14.049 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: waiting osd.0 seq 21474836511 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836511 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:39:14.064 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836511 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383683 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383683 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: waiting osd.1 seq 343597383683 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743434 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743434 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743434 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743434' 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: waiting osd.2 seq 296352743434 2026-02-20T16:39:14.065 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:39:14.754 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 2helpers.sh:2267: flush_pg_stats: echo 3-343597383682 2026-02-20T16:39:14.754 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.754 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:39:14.754 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383682 2026-02-20T16:39:14.754 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383682' 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: waiting osd.3 seq 343597383682 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:39:14.755 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:39:14.756 INFO:tasks.workunit.client.0.trial013.stderr:163664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:39:14.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:39:14.783 INFO:tasks.workunit.client.0.trial013.stderr:96352743434 -lt 296352743434 2026-02-20T16:39:14.783 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:14.783 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-343597383683 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383683 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383683' 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: waiting osd.3 seq 343597383683 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:14.784 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:39:14.785 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:14.786 INFO:tasks.workunit.client.0.trial013.stderr:163661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:14.786 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:39:14.786 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:39:14.786 INFO:tasks.workunit.client.0.trial013.stderr:163661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:39:14.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 163660 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:39:14.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:39:14.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:39:14.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:39:14.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:39:14.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:39:14.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:39:14.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:39:14.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:39:14.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:39:14.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:39:14.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:39:14.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:39:14.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:39:14.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:39:14.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:39:14.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:39:14.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:39:14.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:39:14.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:39:14.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:39:14.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:39:14.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:39:14.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:39:14.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:39:14.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:39:14.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:39:14.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:39:14.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:39:14.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:14.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:14.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:39:14.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:39:14.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:39:14.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:39:14.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:39:14.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:39:14.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:39:14.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:39:14.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:39:14.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:39:14.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:39:14.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:39:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:39:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:39:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:39:14.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:39:14.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:39:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:39:14.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:39:14.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:39:14.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:14.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:14.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:39:14.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:39:14.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:39:14.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:39:14.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:39:14.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:14.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:14.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:39:14.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:39:14.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:39:14.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:901: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=false 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:39:14.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:39:14.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:39:14.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:39:14.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:14.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:14.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:14.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:14.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:14.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:14.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:39:15.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:39:15.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:39:15.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:39:15.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:39:15.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:39:15.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:39:15.024 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:39:15.024 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:39:15.024 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:39:15.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:39:15.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:39:15.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:39:15.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:39:15.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:39:15.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:39:15.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:39:15.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-20T16:39:15.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:39:15.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:39:15.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:39:15.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:39:15.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:39:15.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:39:15.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:39:15.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:15.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:15.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:15.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:15.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:15.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:15.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:39:15.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:39:15.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-20T16:39:15.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:15.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-20T16:39:15.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:39:15.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:15.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:15.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:39:15.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:15.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c8b65084-3792-41b5-938e-4f3c5ee465ba 2026-02-20T16:39:15.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c8b65084-3792-41b5-938e-4f3c5ee465ba' 2026-02-20T16:39:15.268 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 c8b65084-3792-41b5-938e-4f3c5ee465ba 2026-02-20T16:39:15.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:15.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzjphph+t1EBAAFcTq0LC2udQR7nmIElnmSg== 2026-02-20T16:39:15.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzjphph+t1EBAAFcTq0LC2udQR7nmIElnmSg=="}' 2026-02-20T16:39:15.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c8b65084-3792-41b5-938e-4f3c5ee465ba -i td/osd-scrub-repair/0/new.json 2026-02-20T16:39:15.367 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:15.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:39:15.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzjphph+t1EBAAFcTq0LC2udQR7nmIElnmSg== --osd-uuid c8b65084-3792-41b5-938e-4f3c5ee465ba 2026-02-20T16:39:15.391 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:15.385+0000 7f8d79125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:15.393 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:15.389+0000 7f8d79125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:15.393 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:15.389+0000 7f8d79125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:15.394 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:15.389+0000 7f8d79125a40 -1 bdev(0x55576bb96c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:15.394 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:15.389+0000 7f8d79125a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:39:16.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:39:16.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:16.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:39:16.070 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:39:16.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:16.176 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:39:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:39:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:16.201 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:16.197+0000 7fc724685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:16.203 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:16.197+0000 7fc724685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:16.204 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:16.201+0000 7fc724685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:39:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:39:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:16.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:16.271 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:16.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:16.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:16.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:39:16.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:16.778 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:16.773+0000 7fc724685a40 -1 Falling back to public interface 2026-02-20T16:39:17.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:17.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:17.365 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:17.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:17.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:39:17.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:17.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:17.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:17.693+0000 7fc724685a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:39:18.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:18.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:18.500 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:18.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:18.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:18.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:39:18.663 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1195694917,v1:127.0.0.1:6801/1195694917] [v2:127.0.0.1:6802/1195694917,v1:127.0.0.1:6803/1195694917] exists,up c8b65084-3792-41b5-938e-4f3c5ee465ba 2026-02-20T16:39:18.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:18.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:18.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:18.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:18.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-20T16:39:18.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:18.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:18.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:18.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bc91367f-c971-4962-89d5-5e58aa6b0eda 2026-02-20T16:39:18.693 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 bc91367f-c971-4962-89d5-5e58aa6b0eda 2026-02-20T16:39:18.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 bc91367f-c971-4962-89d5-5e58aa6b0eda' 2026-02-20T16:39:18.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:18.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2jphppNJzKBAAihrAO38vbZkRdwSxIE8Buw== 2026-02-20T16:39:18.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2jphppNJzKBAAihrAO38vbZkRdwSxIE8Buw=="}' 2026-02-20T16:39:18.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bc91367f-c971-4962-89d5-5e58aa6b0eda -i td/osd-scrub-repair/1/new.json 2026-02-20T16:39:18.844 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:18.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:39:18.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2jphppNJzKBAAihrAO38vbZkRdwSxIE8Buw== --osd-uuid bc91367f-c971-4962-89d5-5e58aa6b0eda 2026-02-20T16:39:18.871 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:18.865+0000 7fcf53633a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:18.872 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:18.869+0000 7fcf53633a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:18.873 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:18.869+0000 7fcf53633a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:18.874 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:18.869+0000 7fcf53633a40 -1 bdev(0x55a947bc8c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:18.874 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:18.869+0000 7fcf53633a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:39:19.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:39:19.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:19.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:39:19.564 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:39:19.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:19.765 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:39:19.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:39:19.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:19.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:19.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:19.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:19.789 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:19.785+0000 7fe043637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:19.790 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:19.785+0000 7fe043637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:19.791 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:19.785+0000 7fe043637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:19.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:39:19.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:19.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:39:19.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:19.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:19.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:19.933 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:19.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:19.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:19.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:39:20.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:20.269 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:20.265+0000 7fe043637a40 -1 Falling back to public interface 2026-02-20T16:39:20.935 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:20.929+0000 7fe043637a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:39:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:21.093 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:21.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:21.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:21.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:39:21.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:22.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:22.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:22.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:22.260 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:22.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:22.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:39:22.418 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/208406077,v1:127.0.0.1:6811/208406077] [v2:127.0.0.1:6812/208406077,v1:127.0.0.1:6813/208406077] exists,up bc91367f-c971-4962-89d5-5e58aa6b0eda 2026-02-20T16:39:22.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:39:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:22.423 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:22.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:22.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:22.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:22.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:22.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:39:22.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:22.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c513a186-7d49-4c1b-992a-ade2de0bee3d 2026-02-20T16:39:22.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c513a186-7d49-4c1b-992a-ade2de0bee3d' 2026-02-20T16:39:22.426 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 c513a186-7d49-4c1b-992a-ade2de0bee3d 2026-02-20T16:39:22.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:22.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6jphp5zvlGRAAfexbMV2U/doFVWqoQsiOyw== 2026-02-20T16:39:22.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6jphp5zvlGRAAfexbMV2U/doFVWqoQsiOyw=="}' 2026-02-20T16:39:22.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c513a186-7d49-4c1b-992a-ade2de0bee3d -i td/osd-scrub-repair/2/new.json 2026-02-20T16:39:22.598 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:22.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:39:22.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6jphp5zvlGRAAfexbMV2U/doFVWqoQsiOyw== --osd-uuid c513a186-7d49-4c1b-992a-ade2de0bee3d 2026-02-20T16:39:22.633 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:22.629+0000 7f9128a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:22.634 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:22.629+0000 7f9128a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:22.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:22.629+0000 7f9128a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:22.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:22.633+0000 7f9128a92a40 -1 bdev(0x56210127ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:22.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:22.633+0000 7f9128a92a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:39:23.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:39:23.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:23.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:39:23.281 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:39:23.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:23.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:39:23.488 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:39:23.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:23.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:23.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:23.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:23.513 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:23.509+0000 7fda8c26da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:23.515 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:23.509+0000 7fda8c26da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:23.516 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:23.509+0000 7fda8c26da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:23.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:23.647 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:39:23.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:23.938 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:23.933+0000 7fda8c26da40 -1 Falling back to public interface 2026-02-20T16:39:24.598 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:24.593+0000 7fda8c26da40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:39:24.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:24.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:24.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:24.811 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:24.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:24.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:39:24.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:25.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:25.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:25.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:25.982 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:25.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:25.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:39:26.144 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/115543907,v1:127.0.0.1:6819/115543907] [v2:127.0.0.1:6820/115543907,v1:127.0.0.1:6821/115543907] exists,up c513a186-7d49-4c1b-992a-ade2de0bee3d 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:26.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:26.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:39:26.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:26.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:39:26.152 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:39:26.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a' 2026-02-20T16:39:26.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:26.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+jphpmr2aCRAA5G0jweARJe56A1EaZHehwA== 2026-02-20T16:39:26.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+jphpmr2aCRAA5G0jweARJe56A1EaZHehwA=="}' 2026-02-20T16:39:26.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a -i td/osd-scrub-repair/3/new.json 2026-02-20T16:39:26.317 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T16:39:26.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T16:39:26.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+jphpmr2aCRAA5G0jweARJe56A1EaZHehwA== --osd-uuid 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:39:26.353 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:26.349+0000 7f31c5a5fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:26.354 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:26.349+0000 7f31c5a5fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:26.356 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:26.349+0000 7f31c5a5fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:26.356 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:26.353+0000 7f31c5a5fa40 -1 bdev(0x557a742dac00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:26.356 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:26.353+0000 7f31c5a5fa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T16:39:26.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T16:39:26.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:26.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T16:39:26.976 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T16:39:26.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T16:39:27.182 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T16:39:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:27.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:27.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:27.201+0000 7f69dd2d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:27.208 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:27.205+0000 7f69dd2d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:27.209 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:27.205+0000 7f69dd2d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:27.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:27.341 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:27.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:27.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:39:27.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:27.622 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:27.617+0000 7f69dd2d2a40 -1 Falling back to public interface 2026-02-20T16:39:28.218 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:28.213+0000 7f69dd2d2a40 -1 osd.3 0 log_to_monitors true 2026-02-20T16:39:28.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:28.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:28.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:28.501 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:28.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:28.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:39:28.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:29.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:29.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:29.666 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:29.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:29.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:29.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:39:29.826 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4073285626,v1:127.0.0.1:6827/4073285626] [v2:127.0.0.1:6828/4073285626,v1:127.0.0.1:6829/4073285626] exists,up 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:39:29.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:29.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:29.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:29.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:29.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:29.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:29.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:29.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:29.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-20T16:39:29.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:29.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4aae6385-7ee2-4803-98f1-c3ba375ae945 2026-02-20T16:39:29.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 4aae6385-7ee2-4803-98f1-c3ba375ae945' 2026-02-20T16:39:29.834 INFO:tasks.workunit.client.0.trial013.stdout:add osd4 4aae6385-7ee2-4803-98f1-c3ba375ae945 2026-02-20T16:39:29.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:29.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBjphpHjIwMhAAjA0hpXh07qDE4qfsW4DYSg== 2026-02-20T16:39:29.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBjphpHjIwMhAAjA0hpXh07qDE4qfsW4DYSg=="}' 2026-02-20T16:39:29.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4aae6385-7ee2-4803-98f1-c3ba375ae945 -i td/osd-scrub-repair/4/new.json 2026-02-20T16:39:30.005 INFO:tasks.workunit.client.0.trial013.stdout:4 2026-02-20T16:39:30.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-20T16:39:30.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBjphpHjIwMhAAjA0hpXh07qDE4qfsW4DYSg== --osd-uuid 4aae6385-7ee2-4803-98f1-c3ba375ae945 2026-02-20T16:39:30.033 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.029+0000 7f52b762ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:30.034 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.029+0000 7f52b762ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:30.035 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.029+0000 7f52b762ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:30.036 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.029+0000 7f52b762ca40 -1 bdev(0x55f78554cc00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:30.036 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.029+0000 7f52b762ca40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-20T16:39:30.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-20T16:39:30.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:30.644 INFO:tasks.workunit.client.0.trial013.stdout:adding osd4 key to auth repository 2026-02-20T16:39:30.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-20T16:39:30.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:30.841 INFO:tasks.workunit.client.0.trial013.stdout:start osd.4 2026-02-20T16:39:30.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-20T16:39:30.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:30.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:30.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:30.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:30.865 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.861+0000 7f568da1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:30.866 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.861+0000 7f568da1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:30.867 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:30.861+0000 7f568da1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:31.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-20T16:39:31.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:31.006 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:31.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:31.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:39:31.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:31.305 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:31.301+0000 7f568da1aa40 -1 Falling back to public interface 2026-02-20T16:39:31.957 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:31.953+0000 7f568da1aa40 -1 osd.4 0 log_to_monitors true 2026-02-20T16:39:32.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:32.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:32.167 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:39:32.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:33.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:33.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:33.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:33.368 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:33.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:33.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:39:33.526 INFO:tasks.workunit.client.0.trial013.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3030673340,v1:127.0.0.1:6835/3030673340] [v2:127.0.0.1:6836/3030673340,v1:127.0.0.1:6837/3030673340] exists,up 4aae6385-7ee2-4803-98f1-c3ba375ae945 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-20T16:39:33.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:33.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:33.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:39:33.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:33.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:39:33.533 INFO:tasks.workunit.client.0.trial013.stdout:add osd5 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:39:33.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 69574da6-9125-48f9-bca2-c80d60bd439c' 2026-02-20T16:39:33.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:33.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFjphpkyVTIBAApGHRFY13I/lRz8xiIHLy8Q== 2026-02-20T16:39:33.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFjphpkyVTIBAApGHRFY13I/lRz8xiIHLy8Q=="}' 2026-02-20T16:39:33.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69574da6-9125-48f9-bca2-c80d60bd439c -i td/osd-scrub-repair/5/new.json 2026-02-20T16:39:33.710 INFO:tasks.workunit.client.0.trial013.stdout:5 2026-02-20T16:39:33.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-20T16:39:33.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFjphpkyVTIBAApGHRFY13I/lRz8xiIHLy8Q== --osd-uuid 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:39:33.738 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:33.733+0000 7f8339844a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:33.739 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:33.733+0000 7f8339844a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:33.740 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:33.737+0000 7f8339844a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:33.741 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:33.737+0000 7f8339844a40 -1 bdev(0x5585f4376c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:33.741 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:33.737+0000 7f8339844a40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-20T16:39:34.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-20T16:39:34.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:34.412 INFO:tasks.workunit.client.0.trial013.stdout:adding osd5 key to auth repository 2026-02-20T16:39:34.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-20T16:39:34.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stdout:start osd.5 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:34.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:34.625 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:34.621+0000 7f7f2fc17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:34.627 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:34.621+0000 7f7f2fc17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:34.628 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:34.625+0000 7f7f2fc17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:34.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:34.770 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:34.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:34.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:34.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:39:34.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:35.033 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:35.029+0000 7f7f2fc17a40 -1 Falling back to public interface 2026-02-20T16:39:35.701 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:35.697+0000 7f7f2fc17a40 -1 osd.5 0 log_to_monitors true 2026-02-20T16:39:35.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:35.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:35.928 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:35.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:35.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:35.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:39:36.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:37.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:39:37.247 INFO:tasks.workunit.client.0.trial013.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3385970988,v1:127.0.0.1:6843/3385970988] [v2:127.0.0.1:6844/3385970988,v1:127.0.0.1:6845/3385970988] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:39:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-20T16:39:37.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:37.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-20T16:39:37.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:37.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d9cdfcd3-826d-45dd-9b90-8c1fff667804 2026-02-20T16:39:37.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 d9cdfcd3-826d-45dd-9b90-8c1fff667804' 2026-02-20T16:39:37.255 INFO:tasks.workunit.client.0.trial013.stdout:add osd6 d9cdfcd3-826d-45dd-9b90-8c1fff667804 2026-02-20T16:39:37.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:37.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJjphp5D63DxAAPPfO4pyaFNr6+3IRBh1WSg== 2026-02-20T16:39:37.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJjphp5D63DxAAPPfO4pyaFNr6+3IRBh1WSg=="}' 2026-02-20T16:39:37.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d9cdfcd3-826d-45dd-9b90-8c1fff667804 -i td/osd-scrub-repair/6/new.json 2026-02-20T16:39:37.416 INFO:tasks.workunit.client.0.trial013.stdout:6 2026-02-20T16:39:37.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-20T16:39:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJjphp5D63DxAAPPfO4pyaFNr6+3IRBh1WSg== --osd-uuid d9cdfcd3-826d-45dd-9b90-8c1fff667804 2026-02-20T16:39:37.451 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:37.445+0000 7f4261519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:37.453 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:37.449+0000 7f4261519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:37.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:37.449+0000 7f4261519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:37.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:37.449+0000 7f4261519a40 -1 bdev(0x56451955ec00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:37.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:37.449+0000 7f4261519a40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-20T16:39:38.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-20T16:39:38.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:38.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-20T16:39:38.149 INFO:tasks.workunit.client.0.trial013.stdout:adding osd6 key to auth repository 2026-02-20T16:39:38.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:38.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-20T16:39:38.356 INFO:tasks.workunit.client.0.trial013.stdout:start osd.6 2026-02-20T16:39:38.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:38.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:38.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:38.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:38.381 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:38.377+0000 7fda07baea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:38.382 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:38.377+0000 7fda07baea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:38.383 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:38.377+0000 7fda07baea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:38.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:38.513 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:38.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:38.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:38.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:39:38.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:38.769 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:38.765+0000 7fda07baea40 -1 Falling back to public interface 2026-02-20T16:39:39.390 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:39.385+0000 7fda07baea40 -1 osd.6 0 log_to_monitors true 2026-02-20T16:39:39.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:39.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:39.672 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:39.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:39.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:39.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:39:39.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:40.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:40.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:40.833 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:40.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:39:40.991 INFO:tasks.workunit.client.0.trial013.stdout:osd.6 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3208949080,v1:127.0.0.1:6851/3208949080] [v2:127.0.0.1:6852/3208949080,v1:127.0.0.1:6853/3208949080] exists,up d9cdfcd3-826d-45dd-9b90-8c1fff667804 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:40.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-20T16:39:40.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:40.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-20T16:39:40.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:40.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b22bca04-3a36-42b9-ba85-16cbdb60ea40 2026-02-20T16:39:40.999 INFO:tasks.workunit.client.0.trial013.stdout:add osd7 b22bca04-3a36-42b9-ba85-16cbdb60ea40 2026-02-20T16:39:41.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 b22bca04-3a36-42b9-ba85-16cbdb60ea40' 2026-02-20T16:39:41.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:41.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNjphpB5V+ABAAL7hq4foNUBRw7cTzhB81JQ== 2026-02-20T16:39:41.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNjphpB5V+ABAAL7hq4foNUBRw7cTzhB81JQ=="}' 2026-02-20T16:39:41.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b22bca04-3a36-42b9-ba85-16cbdb60ea40 -i td/osd-scrub-repair/7/new.json 2026-02-20T16:39:41.169 INFO:tasks.workunit.client.0.trial013.stdout:7 2026-02-20T16:39:41.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-20T16:39:41.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNjphpB5V+ABAAL7hq4foNUBRw7cTzhB81JQ== --osd-uuid b22bca04-3a36-42b9-ba85-16cbdb60ea40 2026-02-20T16:39:41.194 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:41.189+0000 7f2546ce8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:41.196 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:41.193+0000 7f2546ce8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:41.197 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:41.193+0000 7f2546ce8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:41.197 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:41.193+0000 7f2546ce8a40 -1 bdev(0x562005c60c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:41.197 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:41.193+0000 7f2546ce8a40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-20T16:39:41.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-20T16:39:41.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:41.892 INFO:tasks.workunit.client.0.trial013.stdout:adding osd7 key to auth repository 2026-02-20T16:39:41.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-20T16:39:41.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:42.088 INFO:tasks.workunit.client.0.trial013.stdout:start osd.7 2026-02-20T16:39:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-20T16:39:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:42.112 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:42.109+0000 7f1809a1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:42.114 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:42.109+0000 7f1809a1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:42.115 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:42.109+0000 7f1809a1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:42.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-20T16:39:42.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:42.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-20T16:39:42.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:42.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:42.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:42.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:42.248 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:42.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:42.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:39:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:42.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:42.505+0000 7f1809a1ba40 -1 Falling back to public interface 2026-02-20T16:39:43.189 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:43.185+0000 7f1809a1ba40 -1 osd.7 0 log_to_monitors true 2026-02-20T16:39:43.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:43.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:43.401 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:43.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:43.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:43.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:39:43.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:44.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:44.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:44.577 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:44.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:44.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:44.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:39:44.734 INFO:tasks.workunit.client.0.trial013.stdout:osd.7 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3684376224,v1:127.0.0.1:6859/3684376224] [v2:127.0.0.1:6860/3684376224,v1:127.0.0.1:6861/3684376224] exists,up b22bca04-3a36-42b9-ba85-16cbdb60ea40 2026-02-20T16:39:44.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:44.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:44.739 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:44.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:44.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:44.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:44.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-20T16:39:44.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:44.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=320c85cf-be90-40b6-860b-971bb8397f79 2026-02-20T16:39:44.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 320c85cf-be90-40b6-860b-971bb8397f79' 2026-02-20T16:39:44.742 INFO:tasks.workunit.client.0.trial013.stdout:add osd8 320c85cf-be90-40b6-860b-971bb8397f79 2026-02-20T16:39:44.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:44.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQjphpKEu+LBAACDkzfgXiEJble9ofCO0D/Q== 2026-02-20T16:39:44.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQjphpKEu+LBAACDkzfgXiEJble9ofCO0D/Q=="}' 2026-02-20T16:39:44.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 320c85cf-be90-40b6-860b-971bb8397f79 -i td/osd-scrub-repair/8/new.json 2026-02-20T16:39:44.918 INFO:tasks.workunit.client.0.trial013.stdout:8 2026-02-20T16:39:44.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-20T16:39:44.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQjphpKEu+LBAACDkzfgXiEJble9ofCO0D/Q== --osd-uuid 320c85cf-be90-40b6-860b-971bb8397f79 2026-02-20T16:39:44.944 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:44.941+0000 7f5c45c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:44.946 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:44.941+0000 7f5c45c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:44.947 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:44.941+0000 7f5c45c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:44.947 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:44.941+0000 7f5c45c07a40 -1 bdev(0x556e21b3ec00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:44.947 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:44.941+0000 7f5c45c07a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-20T16:39:45.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-20T16:39:45.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:45.611 INFO:tasks.workunit.client.0.trial013.stdout:adding osd8 key to auth repository 2026-02-20T16:39:45.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-20T16:39:45.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:45.815 INFO:tasks.workunit.client.0.trial013.stdout:start osd.8 2026-02-20T16:39:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-20T16:39:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:45.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:45.837+0000 7fdd7256ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:45.842 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:45.837+0000 7fdd7256ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:45.843 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:45.837+0000 7fdd7256ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:45.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:45.979 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:45.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:45.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:45.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:39:46.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:46.217 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:46.213+0000 7fdd7256ca40 -1 Falling back to public interface 2026-02-20T16:39:46.827 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:46.821+0000 7fdd7256ca40 -1 osd.8 0 log_to_monitors true 2026-02-20T16:39:47.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:47.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:47.136 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:47.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:47.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:47.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:39:47.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:48.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:48.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:48.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:48.293 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:48.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:48.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:39:48.448 INFO:tasks.workunit.client.0.trial013.stdout:osd.8 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/39252591,v1:127.0.0.1:6867/39252591] [v2:127.0.0.1:6868/39252591,v1:127.0.0.1:6869/39252591] exists,up 320c85cf-be90-40b6-860b-971bb8397f79 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-20T16:39:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:39:48.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:39:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:39:48.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:39:48.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:39:48.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:39:48.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:39:48.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:39:48.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:39:48.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:39:48.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 1da8e7f8-9f5e-4312-9f00-c7d153459349' 2026-02-20T16:39:48.456 INFO:tasks.workunit.client.0.trial013.stdout:add osd9 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:39:48.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:39:48.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUjphpCZO+GxAAW7x7DGT+y79/GXT5BHdTBQ== 2026-02-20T16:39:48.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUjphpCZO+GxAAW7x7DGT+y79/GXT5BHdTBQ=="}' 2026-02-20T16:39:48.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1da8e7f8-9f5e-4312-9f00-c7d153459349 -i td/osd-scrub-repair/9/new.json 2026-02-20T16:39:48.624 INFO:tasks.workunit.client.0.trial013.stdout:9 2026-02-20T16:39:48.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-20T16:39:48.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUjphpCZO+GxAAW7x7DGT+y79/GXT5BHdTBQ== --osd-uuid 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:39:48.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:48.645+0000 7f21348bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:48.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:48.645+0000 7f21348bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:48.652 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:48.649+0000 7f21348bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:48.653 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:48.649+0000 7f21348bfa40 -1 bdev(0x559653f20c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-20T16:39:48.653 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:48.649+0000 7f21348bfa40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-20T16:39:49.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-20T16:39:49.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:39:49.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-20T16:39:49.356 INFO:tasks.workunit.client.0.trial013.stdout:adding osd9 key to auth repository 2026-02-20T16:39:49.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:39:49.560 INFO:tasks.workunit.client.0.trial013.stdout:start osd.9 2026-02-20T16:39:49.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-20T16:39:49.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:39:49.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:39:49.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:39:49.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:39:49.584 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:49.581+0000 7f26b7d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:49.585 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:49.581+0000 7f26b7d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:49.587 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:49.581+0000 7f26b7d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:39:49.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-20T16:39:49.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:39:49.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:39:49.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:39:49.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:39:49.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:49.724 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:39:49.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:39:49.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:49.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:39:49.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:49.997 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:49.993+0000 7f26b7d2aa40 -1 Falling back to public interface 2026-02-20T16:39:50.610 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:39:50.605+0000 7f26b7d2aa40 -1 osd.9 0 log_to_monitors true 2026-02-20T16:39:50.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:50.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:50.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:39:50.886 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:39:50.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:50.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:39:51.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:39:52.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:39:52.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:39:52.048 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:39:52.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:39:52.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:39:52.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:39:52.200 INFO:tasks.workunit.client.0.trial013.stdout:osd.9 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1597750875,v1:127.0.0.1:6875/1597750875] [v2:127.0.0.1:6876/1597750875,v1:127.0.0.1:6877/1597750875] exists,up 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:39:52.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:39:52.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:39:52.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:39:52.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-20T16:39:52.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:39:52.350 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:39:52.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:39:52.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:39:52.546 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:39:52.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:39:53.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:39:53.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-20T16:39:53.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:39:53.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:39:53.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:39:53.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:39:53.842 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:39:53.842 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:39:53.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:39:53.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:39:53.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:39:53.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:39:53.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:39:53.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:39:53.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:39:53.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:39:53.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:39:54.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:39:54.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:39:54.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-20T16:39:54.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-20T16:39:54.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-20T16:39:54.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:39:54.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-20T16:39:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-20T16:39:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-20T16:39:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:39:54.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-20T16:39:54.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-20T16:39:54.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855' 2026-02-20T16:39:54.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:39:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411335 2026-02-20T16:39:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411335 2026-02-20T16:39:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335' 2026-02-20T16:39:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:39:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247814 2026-02-20T16:39:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247814 2026-02-20T16:39:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814' 2026-02-20T16:39:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:39:54.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-20T16:39:54.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-20T16:39:54.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814 5-115964116997' 2026-02-20T16:39:54.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:39:54.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-20T16:39:54.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-20T16:39:54.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814 5-115964116997 6-137438953476' 2026-02-20T16:39:54.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:39:54.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822660 2026-02-20T16:39:54.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822660 2026-02-20T16:39:54.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814 5-115964116997 6-137438953476 7-154618822660' 2026-02-20T16:39:54.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:39:54.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659139 2026-02-20T16:39:54.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659139 2026-02-20T16:39:54.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814 5-115964116997 6-137438953476 7-154618822660 8-176093659139' 2026-02-20T16:39:54.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:39:54.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:39:54.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495618 2026-02-20T16:39:54.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495618 2026-02-20T16:39:54.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411335 4-98784247814 5-115964116997 6-137438953476 7-154618822660 8-176093659139 9-197568495618' 2026-02-20T16:39:54.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:54.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-20T16:39:54.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:54.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:39:54.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-20T16:39:54.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:54.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-20T16:39:54.840 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836489 2026-02-20T16:39:54.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-20T16:39:54.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:55.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-20T16:39:55.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:39:56.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:39:56.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:39:56.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-20T16:39:56.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:56.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-20T16:39:56.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:56.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:39:56.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-20T16:39:56.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:56.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-20T16:39:56.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-20T16:39:56.171 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705672 2026-02-20T16:39:56.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:39:56.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-20T16:39:56.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:56.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574855 2026-02-20T16:39:56.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:56.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:39:56.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574855 2026-02-20T16:39:56.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:56.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574855 2026-02-20T16:39:56.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2026-02-20T16:39:56.346 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574855 2026-02-20T16:39:56.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:39:56.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-20T16:39:56.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:56.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411335 2026-02-20T16:39:56.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:56.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:39:56.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411335 2026-02-20T16:39:56.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:56.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411335 2026-02-20T16:39:56.505 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 77309411335 2026-02-20T16:39:56.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411335' 2026-02-20T16:39:56.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:39:56.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-20T16:39:56.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:56.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247814 2026-02-20T16:39:56.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:56.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:39:56.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247814 2026-02-20T16:39:56.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:56.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247814 2026-02-20T16:39:56.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2026-02-20T16:39:56.679 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.4 seq 98784247814 2026-02-20T16:39:56.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:39:56.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-20T16:39:56.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:56.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116997 2026-02-20T16:39:56.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:56.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:39:56.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116997 2026-02-20T16:39:56.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:56.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116997 2026-02-20T16:39:56.842 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.5 seq 115964116997 2026-02-20T16:39:56.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116997' 2026-02-20T16:39:56.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:39:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-20T16:39:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:57.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953476 2026-02-20T16:39:57.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:57.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:39:57.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953476 2026-02-20T16:39:57.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:57.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-20T16:39:57.017 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.6 seq 137438953476 2026-02-20T16:39:57.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953476' 2026-02-20T16:39:57.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:39:57.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953476 2026-02-20T16:39:57.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:57.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822660 2026-02-20T16:39:57.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:57.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:39:57.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822660 2026-02-20T16:39:57.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:57.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822660 2026-02-20T16:39:57.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822660' 2026-02-20T16:39:57.188 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.7 seq 154618822660 2026-02-20T16:39:57.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:39:57.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822660 -lt 154618822660 2026-02-20T16:39:57.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:57.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659139 2026-02-20T16:39:57.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:57.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:39:57.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659139 2026-02-20T16:39:57.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:57.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659139 2026-02-20T16:39:57.362 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.8 seq 176093659139 2026-02-20T16:39:57.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659139' 2026-02-20T16:39:57.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:39:57.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659139 -lt 176093659139 2026-02-20T16:39:57.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:39:57.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495618 2026-02-20T16:39:57.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:39:57.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:39:57.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495618 2026-02-20T16:39:57.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:39:57.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495618 2026-02-20T16:39:57.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495618' 2026-02-20T16:39:57.525 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.9 seq 197568495618 2026-02-20T16:39:57.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:39:57.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495619 -lt 197568495618 2026-02-20T16:39:57.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:39:57.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:57.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:39:57.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:39:57.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:39:57.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:39:57.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:39:57.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:39:57.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:39:57.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:39:58.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:39:58.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:39:58.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:39:58.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:39:58.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:39:58.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-20T16:39:58.443 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:39:58.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:39:58.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:39:58.749 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:39:58.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:39:59.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:39:59.763 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:39:59.763 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:39:59.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:39:59.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:39:59.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:39:59.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:39:59.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:39:59.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:39:59.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:39:59.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:39:59.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:00.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:00.038 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:00.038 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:00.038 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:00.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:40:00.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:40:00.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:40:00.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:00.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-20T16:40:00.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-20T16:40:00.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2026-02-20T16:40:00.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-20T16:40:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-20T16:40:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857' 2026-02-20T16:40:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:00.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-20T16:40:00.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-20T16:40:00.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337' 2026-02-20T16:40:00.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:00.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247816 2026-02-20T16:40:00.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247816 2026-02-20T16:40:00.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816' 2026-02-20T16:40:00.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:00.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-20T16:40:00.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-20T16:40:00.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816 5-115964116999' 2026-02-20T16:40:00.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:00.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953478 2026-02-20T16:40:00.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953478 2026-02-20T16:40:00.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816 5-115964116999 6-137438953478' 2026-02-20T16:40:00.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:00.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822662 2026-02-20T16:40:00.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822662 2026-02-20T16:40:00.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816 5-115964116999 6-137438953478 7-154618822662' 2026-02-20T16:40:00.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:00.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659141 2026-02-20T16:40:00.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659141 2026-02-20T16:40:00.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816 5-115964116999 6-137438953478 7-154618822662 8-176093659141' 2026-02-20T16:40:00.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:00.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:00.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495620 2026-02-20T16:40:00.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495620 2026-02-20T16:40:00.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-98784247816 5-115964116999 6-137438953478 7-154618822662 8-176093659141 9-197568495620' 2026-02-20T16:40:00.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:40:00.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:00.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:00.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:40:00.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:00.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:40:00.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:40:00.772 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T16:40:00.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:00.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:40:00.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:01.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:01.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:02.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:40:02.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-20T16:40:02.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:02.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705674 2026-02-20T16:40:02.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705674 2026-02-20T16:40:02.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-20T16:40:02.103 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705674 2026-02-20T16:40:02.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:02.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-20T16:40:02.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-20T16:40:02.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:02.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-20T16:40:02.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-20T16:40:02.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-20T16:40:02.273 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574857 2026-02-20T16:40:02.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:02.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574857 2026-02-20T16:40:02.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411337 2026-02-20T16:40:02.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:02.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411337 2026-02-20T16:40:02.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411337 2026-02-20T16:40:02.439 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 77309411337 2026-02-20T16:40:02.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2026-02-20T16:40:02.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:02.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-20T16:40:02.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247816 2026-02-20T16:40:02.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:02.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247816 2026-02-20T16:40:02.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247816 2026-02-20T16:40:02.611 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.4 seq 98784247816 2026-02-20T16:40:02.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247816' 2026-02-20T16:40:02.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:02.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247816 -lt 98784247816 2026-02-20T16:40:02.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116999 2026-02-20T16:40:02.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:02.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116999 2026-02-20T16:40:02.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-20T16:40:02.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116999' 2026-02-20T16:40:02.774 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.5 seq 115964116999 2026-02-20T16:40:02.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:02.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-20T16:40:02.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:02.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953478 2026-02-20T16:40:02.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:02.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:02.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953478 2026-02-20T16:40:02.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:02.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953478 2026-02-20T16:40:02.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953478' 2026-02-20T16:40:02.946 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.6 seq 137438953478 2026-02-20T16:40:02.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:03.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953479 -lt 137438953478 2026-02-20T16:40:03.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:03.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822662 2026-02-20T16:40:03.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:03.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:03.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822662 2026-02-20T16:40:03.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:03.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822662 2026-02-20T16:40:03.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822662' 2026-02-20T16:40:03.115 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.7 seq 154618822662 2026-02-20T16:40:03.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:03.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822662 -lt 154618822662 2026-02-20T16:40:03.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:03.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659141 2026-02-20T16:40:03.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:03.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:03.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659141 2026-02-20T16:40:03.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:03.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659141 2026-02-20T16:40:03.280 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.8 seq 176093659141 2026-02-20T16:40:03.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659141' 2026-02-20T16:40:03.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:03.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659141 -lt 176093659141 2026-02-20T16:40:03.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:03.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495620 2026-02-20T16:40:03.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:03.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495620 2026-02-20T16:40:03.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:03.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495620 2026-02-20T16:40:03.453 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.9 seq 197568495620 2026-02-20T16:40:03.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495620' 2026-02-20T16:40:03.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:03.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495621 -lt 197568495620 2026-02-20T16:40:03.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:03.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:03.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:03.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:03.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:03.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:03.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:03.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:03.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:03.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:03.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:03.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:03.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:03.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:03.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:40:04.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:40:04.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:40:04.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:40:04.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:40:04.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:40:04.401 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:40:04.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:40:04.603 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:40:04.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:40:04.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:40:04.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:40:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T16:40:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:40:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:40:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:40:04.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:40:04.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-20T16:40:04.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T16:40:04.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-20T16:40:04.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:40:04.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T16:40:04.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:40:04.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:40:04.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-20T16:40:04.955 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:04.955 INFO:tasks.workunit.client.0.trial013.stderr:9 2026-02-20T16:40:04.955 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:40:04.956 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:04.956 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:04.956 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:04.956 INFO:tasks.workunit.client.0.trial013.stderr:7' 2026-02-20T16:40:04.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-20T16:40:04.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:40:04.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:40:04.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-20T16:40:04.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:40:04.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:04.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:04.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:40:04.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:04.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:40:04.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:04.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:04.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:04.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:04.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:05.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:40:05.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:40:05.467 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:40:05.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:05.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:05.593 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:05.588+0000 7f3fa89aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:05.594 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:05.588+0000 7f3fa89aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:05.595 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:05.588+0000 7f3fa89aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:05.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:05.738 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:40:05.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:05.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:40:05.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:06.037 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:06.032+0000 7f3fa89aba40 -1 Falling back to public interface 2026-02-20T16:40:06.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:06.688+0000 7f3fa89aba40 -1 osd.3 62 log_to_monitors true 2026-02-20T16:40:06.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:06.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:06.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:06.897 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:06.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:06.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:40:07.056 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [18,62) [v2:127.0.0.1:6826/1941570186,v1:127.0.0.1:6827/1941570186] [v2:127.0.0.1:6828/1941570186,v1:127.0.0.1:6829/1941570186] exists,up 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:40:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:07.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:07.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:07.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:07.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:40:07.331 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:07.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:40:07.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:40:07.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:40:07.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:07.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-20T16:40:07.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-20T16:40:07.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677' 2026-02-20T16:40:07.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:07.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-20T16:40:07.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-20T16:40:07.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860' 2026-02-20T16:40:07.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-20T16:40:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-20T16:40:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538' 2026-02-20T16:40:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:07.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247818 2026-02-20T16:40:07.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247818 2026-02-20T16:40:07.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818' 2026-02-20T16:40:07.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:07.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117002 2026-02-20T16:40:07.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117002 2026-02-20T16:40:07.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818 5-115964117002' 2026-02-20T16:40:07.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:07.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953481 2026-02-20T16:40:07.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953481 2026-02-20T16:40:07.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818 5-115964117002 6-137438953481' 2026-02-20T16:40:07.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:07.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822664 2026-02-20T16:40:07.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822664 2026-02-20T16:40:07.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818 5-115964117002 6-137438953481 7-154618822664' 2026-02-20T16:40:07.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:07.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659143 2026-02-20T16:40:07.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659143 2026-02-20T16:40:07.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818 5-115964117002 6-137438953481 7-154618822664 8-176093659143' 2026-02-20T16:40:07.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:07.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:08.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495623 2026-02-20T16:40:08.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495623 2026-02-20T16:40:08.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-98784247818 5-115964117002 6-137438953481 7-154618822664 8-176093659143 9-197568495623' 2026-02-20T16:40:08.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:08.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:40:08.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:08.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:08.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:40:08.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:40:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:40:08.067 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:40:08.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:08.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T16:40:08.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:08.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-20T16:40:08.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:08.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:08.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-20T16:40:08.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:08.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-20T16:40:08.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-20T16:40:08.239 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705677 2026-02-20T16:40:08.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:08.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705677 -lt 38654705677 2026-02-20T16:40:08.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:08.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574860 2026-02-20T16:40:08.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:08.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:08.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574860 2026-02-20T16:40:08.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:08.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574860 2026-02-20T16:40:08.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-20T16:40:08.414 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574860 2026-02-20T16:40:08.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:08.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574860 2026-02-20T16:40:08.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:09.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:09.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-20T16:40:09.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:09.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841538 2026-02-20T16:40:09.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:09.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:09.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841538 2026-02-20T16:40:09.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:09.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-20T16:40:09.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841538' 2026-02-20T16:40:09.749 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 283467841538 2026-02-20T16:40:09.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:09.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-20T16:40:09.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:09.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247818 2026-02-20T16:40:09.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:09.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:09.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247818 2026-02-20T16:40:09.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:09.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247818 2026-02-20T16:40:09.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247818' 2026-02-20T16:40:09.924 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247818 2026-02-20T16:40:09.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247819 -lt 98784247818 2026-02-20T16:40:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:10.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117002 2026-02-20T16:40:10.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:10.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:10.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117002 2026-02-20T16:40:10.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117002 2026-02-20T16:40:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117002' 2026-02-20T16:40:10.086 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 115964117002 2026-02-20T16:40:10.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:10.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117002 -lt 115964117002 2026-02-20T16:40:10.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:10.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953481 2026-02-20T16:40:10.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:10.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:10.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953481 2026-02-20T16:40:10.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:10.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953481 2026-02-20T16:40:10.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953481' 2026-02-20T16:40:10.265 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953481 2026-02-20T16:40:10.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953481 -lt 137438953481 2026-02-20T16:40:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:10.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822664 2026-02-20T16:40:10.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:10.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:10.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822664 2026-02-20T16:40:10.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:10.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822664 2026-02-20T16:40:10.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822664' 2026-02-20T16:40:10.435 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 154618822664 2026-02-20T16:40:10.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:10.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822665 -lt 154618822664 2026-02-20T16:40:10.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:10.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659143 2026-02-20T16:40:10.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:10.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:10.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659143 2026-02-20T16:40:10.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659143 2026-02-20T16:40:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659143' 2026-02-20T16:40:10.608 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 176093659143 2026-02-20T16:40:10.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:10.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659144 -lt 176093659143 2026-02-20T16:40:10.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:10.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495623 2026-02-20T16:40:10.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:10.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:10.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495623 2026-02-20T16:40:10.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:10.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495623 2026-02-20T16:40:10.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495623' 2026-02-20T16:40:10.772 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 197568495623 2026-02-20T16:40:10.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:10.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495623 -lt 197568495623 2026-02-20T16:40:10.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:10.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:10.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:11.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:11.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:11.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:11.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:11.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:11.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:11.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:11.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:11.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:11.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:11.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:11.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:40:11.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:40:11.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:40:11.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:11.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:11.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:11.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:11.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:11.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:40:11.954 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:11.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:11.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:11.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:12.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:58.748214+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:12.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:13.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:13.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:13.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:13.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:13.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:13.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:13.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:13.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:58.748214+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:13.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:14.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:14.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:14.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:14.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:14.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:14.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:14.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:14.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:58.748214+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:14.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:15.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:15.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:15.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:15.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:15.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:15.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:15.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:58.748214+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:16.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:16.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:16.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:16.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:16.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:16.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:16.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:39:58.748214+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:16.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:17.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:17.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:17.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:17.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:17.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:17.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:17.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:17.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:12.648524+0000 '>' 2026-02-20T16:39:58.748214+0000 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:18.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:18.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:40:18.310 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:40:18.310 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:40:18.310 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:40:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:40:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:18.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:18.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:40:18.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:40:18.422 INFO:tasks.workunit.client.0.trial013.stderr:start osd.9 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:18.437 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:18.432+0000 7fe81299ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:18.438 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:18.432+0000 7fe81299ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:18.439 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:18.432+0000 7fe81299ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:18.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:18.582 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:40:18.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:18.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:18.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:18.873 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:18.868+0000 7fe81299ba40 -1 Falling back to public interface 2026-02-20T16:40:19.532 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:19.528+0000 7fe81299ba40 -1 osd.9 68 log_to_monitors true 2026-02-20T16:40:19.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:19.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:19.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:19.742 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:19.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:19.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:osd.9 up in weight 1 up_from 71 up_thru 0 down_at 68 last_clean_interval [46,68) [v2:127.0.0.1:6874/3300696493,v1:127.0.0.1:6875/3300696493] [v2:127.0.0.1:6876/3300696493,v1:127.0.0.1:6877/3300696493] exists,up 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:19.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:19.905 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:19.905 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:40:19.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:19.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:19.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:20.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:20.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:20.179 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:20.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-20T16:40:20.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-20T16:40:20.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-20T16:40:20.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705680 2026-02-20T16:40:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705680 2026-02-20T16:40:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680' 2026-02-20T16:40:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:20.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-20T16:40:20.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-20T16:40:20.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863' 2026-02-20T16:40:20.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841541 2026-02-20T16:40:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841541 2026-02-20T16:40:20.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541' 2026-02-20T16:40:20.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:20.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247822 2026-02-20T16:40:20.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247822 2026-02-20T16:40:20.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822' 2026-02-20T16:40:20.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:20.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-20T16:40:20.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-20T16:40:20.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822 5-115964117005' 2026-02-20T16:40:20.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:20.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953485 2026-02-20T16:40:20.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953485 2026-02-20T16:40:20.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822 5-115964117005 6-137438953485' 2026-02-20T16:40:20.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822668 2026-02-20T16:40:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822668 2026-02-20T16:40:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822 5-115964117005 6-137438953485 7-154618822668' 2026-02-20T16:40:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:20.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659147 2026-02-20T16:40:20.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659147 2026-02-20T16:40:20.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822 5-115964117005 6-137438953485 7-154618822668 8-176093659147' 2026-02-20T16:40:20.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:20.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:20.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678018 2026-02-20T16:40:20.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678018 2026-02-20T16:40:20.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-98784247822 5-115964117005 6-137438953485 7-154618822668 8-176093659147 9-304942678018' 2026-02-20T16:40:20.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:20.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-20T16:40:20.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:20.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:20.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-20T16:40:20.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:20.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-20T16:40:20.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-20T16:40:20.911 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836497 2026-02-20T16:40:20.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:21.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-20T16:40:21.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:22.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:22.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:22.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-20T16:40:22.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:22.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705680 2026-02-20T16:40:22.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:22.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:22.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705680 2026-02-20T16:40:22.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:22.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705680 2026-02-20T16:40:22.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-20T16:40:22.245 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705680 2026-02-20T16:40:22.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:22.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-20T16:40:22.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:22.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-20T16:40:22.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:22.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:22.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574863 2026-02-20T16:40:22.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:22.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574863 2026-02-20T16:40:22.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-20T16:40:22.409 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574863 2026-02-20T16:40:22.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:22.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574864 -lt 55834574863 2026-02-20T16:40:22.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:22.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841541 2026-02-20T16:40:22.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:22.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:22.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841541 2026-02-20T16:40:22.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:22.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841541 2026-02-20T16:40:22.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841541' 2026-02-20T16:40:22.584 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 283467841541 2026-02-20T16:40:22.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:22.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841541 -lt 283467841541 2026-02-20T16:40:22.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:22.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247822 2026-02-20T16:40:22.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:22.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:22.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247822 2026-02-20T16:40:22.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:22.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247822 2026-02-20T16:40:22.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247822' 2026-02-20T16:40:22.749 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247822 2026-02-20T16:40:22.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:22.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247822 -lt 98784247822 2026-02-20T16:40:22.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:22.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117005 2026-02-20T16:40:22.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:22.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:22.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117005 2026-02-20T16:40:22.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:22.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117005 2026-02-20T16:40:22.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117005' 2026-02-20T16:40:22.911 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 115964117005 2026-02-20T16:40:22.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:23.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-20T16:40:23.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:23.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953485 2026-02-20T16:40:23.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:23.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:23.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953485 2026-02-20T16:40:23.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:23.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953485 2026-02-20T16:40:23.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953485' 2026-02-20T16:40:23.078 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953485 2026-02-20T16:40:23.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:23.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953485 -lt 137438953485 2026-02-20T16:40:23.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:23.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822668 2026-02-20T16:40:23.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:23.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:23.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822668 2026-02-20T16:40:23.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:23.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822668 2026-02-20T16:40:23.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822668' 2026-02-20T16:40:23.247 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 154618822668 2026-02-20T16:40:23.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:23.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822668 -lt 154618822668 2026-02-20T16:40:23.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:23.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659147 2026-02-20T16:40:23.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:23.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:23.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659147 2026-02-20T16:40:23.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:23.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659147 2026-02-20T16:40:23.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659147' 2026-02-20T16:40:23.426 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 176093659147 2026-02-20T16:40:23.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:23.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659148 -lt 176093659147 2026-02-20T16:40:23.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:23.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678018 2026-02-20T16:40:23.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:23.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:23.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678018 2026-02-20T16:40:23.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678018 2026-02-20T16:40:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678018' 2026-02-20T16:40:23.598 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 304942678018 2026-02-20T16:40:23.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:23.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678018 -lt 304942678018 2026-02-20T16:40:23.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:23.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:23.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:23.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:23.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:23.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:24.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:24.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:24.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:24.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:24.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:24.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:24.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:24.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:40:24.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:24.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:24.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:40:24.820 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:24.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:24.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:24.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:24.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:24.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:start osd.5 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:24.943 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:24.936+0000 7f6115fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:24.944 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:24.940+0000 7f6115fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:24.945 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:24.940+0000 7f6115fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:25.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:40:25.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:25.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:25.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:25.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:25.352+0000 7f6115fa6a40 -1 Falling back to public interface 2026-02-20T16:40:26.043 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:26.036+0000 7f6115fa6a40 -1 osd.5 73 log_to_monitors true 2026-02-20T16:40:26.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:26.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:26.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:26.246 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:26.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:26.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:26.401 INFO:tasks.workunit.client.0.trial013.stderr:osd.5 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [27,73) [v2:127.0.0.1:6842/341863878,v1:127.0.0.1:6843/341863878] [v2:127.0.0.1:6844/341863878,v1:127.0.0.1:6845/341863878] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:26.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:26.403 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:26.403 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:40:26.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:26.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:26.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:26.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:26.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:26.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:26.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:26.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:26.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:26.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:40:26.668 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:40:26.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:26.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:26.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:26.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-20T16:40:26.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-20T16:40:26.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-20T16:40:26.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:26.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:26.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705682 2026-02-20T16:40:26.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705682 2026-02-20T16:40:26.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682' 2026-02-20T16:40:26.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:26.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:26.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574866 2026-02-20T16:40:26.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574866 2026-02-20T16:40:26.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866' 2026-02-20T16:40:26.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:26.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:26.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841544 2026-02-20T16:40:26.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841544 2026-02-20T16:40:26.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544' 2026-02-20T16:40:26.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:26.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:27.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247824 2026-02-20T16:40:27.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247824 2026-02-20T16:40:27.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824' 2026-02-20T16:40:27.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:27.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514498 2026-02-20T16:40:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514498 2026-02-20T16:40:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824 5-326417514498' 2026-02-20T16:40:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:27.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:27.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953487 2026-02-20T16:40:27.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953487 2026-02-20T16:40:27.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824 5-326417514498 6-137438953487' 2026-02-20T16:40:27.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:27.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:27.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822670 2026-02-20T16:40:27.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822670 2026-02-20T16:40:27.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824 5-326417514498 6-137438953487 7-154618822670' 2026-02-20T16:40:27.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:27.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:27.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659149 2026-02-20T16:40:27.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659149 2026-02-20T16:40:27.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824 5-326417514498 6-137438953487 7-154618822670 8-176093659149' 2026-02-20T16:40:27.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:27.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:27.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678020 2026-02-20T16:40:27.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678020 2026-02-20T16:40:27.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-98784247824 5-326417514498 6-137438953487 7-154618822670 8-176093659149 9-304942678020' 2026-02-20T16:40:27.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:27.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-20T16:40:27.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:27.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:27.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-20T16:40:27.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:27.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-20T16:40:27.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-20T16:40:27.400 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836499 2026-02-20T16:40:27.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:27.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-20T16:40:27.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:27.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705682 2026-02-20T16:40:27.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:27.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:27.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705682 2026-02-20T16:40:27.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:27.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705682 2026-02-20T16:40:27.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2026-02-20T16:40:27.561 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705682 2026-02-20T16:40:27.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:27.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705683 -lt 38654705682 2026-02-20T16:40:27.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:27.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574866 2026-02-20T16:40:27.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:27.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:27.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574866 2026-02-20T16:40:27.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574866 2026-02-20T16:40:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574866' 2026-02-20T16:40:27.730 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574866 2026-02-20T16:40:27.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:27.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574866 -lt 55834574866 2026-02-20T16:40:27.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:27.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841544 2026-02-20T16:40:27.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:27.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:27.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841544 2026-02-20T16:40:27.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:27.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841544 2026-02-20T16:40:27.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841544' 2026-02-20T16:40:27.907 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 283467841544 2026-02-20T16:40:27.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:28.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841544 -lt 283467841544 2026-02-20T16:40:28.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247824 2026-02-20T16:40:28.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:28.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247824 2026-02-20T16:40:28.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247824 2026-02-20T16:40:28.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247824' 2026-02-20T16:40:28.075 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247824 2026-02-20T16:40:28.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:28.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247824 -lt 98784247824 2026-02-20T16:40:28.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514498 2026-02-20T16:40:28.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:28.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-326417514498 2026-02-20T16:40:28.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514498 2026-02-20T16:40:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 326417514498' 2026-02-20T16:40:28.251 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 326417514498 2026-02-20T16:40:28.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:28.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514498 -lt 326417514498 2026-02-20T16:40:28.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953487 2026-02-20T16:40:28.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:28.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953487 2026-02-20T16:40:28.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953487 2026-02-20T16:40:28.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953487' 2026-02-20T16:40:28.413 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953487 2026-02-20T16:40:28.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:28.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953487 -lt 137438953487 2026-02-20T16:40:28.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822670 2026-02-20T16:40:28.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:28.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822670 2026-02-20T16:40:28.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822670 2026-02-20T16:40:28.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822670' 2026-02-20T16:40:28.582 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 154618822670 2026-02-20T16:40:28.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:28.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822670 -lt 154618822670 2026-02-20T16:40:28.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659149 2026-02-20T16:40:28.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:28.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659149 2026-02-20T16:40:28.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659149 2026-02-20T16:40:28.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659149' 2026-02-20T16:40:28.750 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 176093659149 2026-02-20T16:40:28.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659149 -lt 176093659149 2026-02-20T16:40:28.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:28.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678020 2026-02-20T16:40:28.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:28.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:28.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678020 2026-02-20T16:40:28.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678020 2026-02-20T16:40:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678020' 2026-02-20T16:40:28.919 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 304942678020 2026-02-20T16:40:28.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:29.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678020 -lt 304942678020 2026-02-20T16:40:29.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:29.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:29.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:29.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:29.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:29.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:29.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:29.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:29.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:29.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:29.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:29.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:29.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:29.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:29.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:29.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:29.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:29.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:29.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:40:29.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:40:29.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:40:29.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:40:29.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:29.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:29.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:29.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:29.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:29.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:40:30.110 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:30.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:30.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:12.648524+0000 '>' 2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:30.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:31.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:31.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:31.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:31.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:31.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:31.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:31.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:31.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:12.648524+0000 '>' 2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:31.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:32.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:12.648524+0000 '>' 2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:32.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:33.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:33.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:33.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:33.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:33.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:33.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:33.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:12.648524+0000 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:33.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:33.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:40:34.135 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:40:34.135 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:40:34.135 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:40:34.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:40:34.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:34.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:34.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:start osd.9 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:40:34.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:40:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:34.247 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:34.240+0000 7f4440906a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:34.248 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:34.244+0000 7f4440906a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:34.249 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:34.244+0000 7f4440906a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:34.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:40:34.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:34.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:40:34.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:34.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:34.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:34.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:34.395 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:40:34.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:34.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:34.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:34.681 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:34.676+0000 7f4440906a40 -1 Falling back to public interface 2026-02-20T16:40:35.314 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:40:35.308+0000 7f4440906a40 -1 osd.9 78 log_to_monitors true 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:osd.9 up in weight 1 up_from 82 up_thru 0 down_at 79 last_clean_interval [71,78) [v2:127.0.0.1:6874/1254049254,v1:127.0.0.1:6875/1254049254] [v2:127.0.0.1:6876/1254049254,v1:127.0.0.1:6877/1254049254] exists,up 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:35.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:35.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:35.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:35.713 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:35.713 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:40:35.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:35.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:35.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:35.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:35.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:35.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:35.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:35.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:35.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:35.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:35.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:36.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-20T16:40:36.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-20T16:40:36.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-20T16:40:36.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705685 2026-02-20T16:40:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705685 2026-02-20T16:40:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685' 2026-02-20T16:40:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:36.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574869 2026-02-20T16:40:36.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574869 2026-02-20T16:40:36.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869' 2026-02-20T16:40:36.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841546 2026-02-20T16:40:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841546 2026-02-20T16:40:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546' 2026-02-20T16:40:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:36.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247827 2026-02-20T16:40:36.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247827 2026-02-20T16:40:36.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827' 2026-02-20T16:40:36.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514501 2026-02-20T16:40:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514501 2026-02-20T16:40:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827 5-326417514501' 2026-02-20T16:40:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:36.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953490 2026-02-20T16:40:36.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953490 2026-02-20T16:40:36.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827 5-326417514501 6-137438953490' 2026-02-20T16:40:36.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:36.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822673 2026-02-20T16:40:36.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822673 2026-02-20T16:40:36.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827 5-326417514501 6-137438953490 7-154618822673' 2026-02-20T16:40:36.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:36.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659152 2026-02-20T16:40:36.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659152 2026-02-20T16:40:36.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827 5-326417514501 6-137438953490 7-154618822673 8-176093659152' 2026-02-20T16:40:36.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:36.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318274 2026-02-20T16:40:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318274 2026-02-20T16:40:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-98784247827 5-326417514501 6-137438953490 7-154618822673 8-176093659152 9-352187318274' 2026-02-20T16:40:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-20T16:40:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:36.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-20T16:40:36.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:36.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-20T16:40:36.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-20T16:40:36.716 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836502 2026-02-20T16:40:36.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:36.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836502 2026-02-20T16:40:36.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:37.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:37.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:38.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-20T16:40:38.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705685 2026-02-20T16:40:38.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:38.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705685 2026-02-20T16:40:38.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705685 2026-02-20T16:40:38.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2026-02-20T16:40:38.042 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705685 2026-02-20T16:40:38.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:38.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705686 -lt 38654705685 2026-02-20T16:40:38.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574869 2026-02-20T16:40:38.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:38.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574869 2026-02-20T16:40:38.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574869 2026-02-20T16:40:38.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574869' 2026-02-20T16:40:38.202 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 55834574869 2026-02-20T16:40:38.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:38.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574869 -lt 55834574869 2026-02-20T16:40:38.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841546 2026-02-20T16:40:38.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:38.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841546 2026-02-20T16:40:38.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841546 2026-02-20T16:40:38.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841546' 2026-02-20T16:40:38.371 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 283467841546 2026-02-20T16:40:38.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:38.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841547 -lt 283467841546 2026-02-20T16:40:38.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247827 2026-02-20T16:40:38.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:38.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247827 2026-02-20T16:40:38.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247827 2026-02-20T16:40:38.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247827' 2026-02-20T16:40:38.542 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247827 2026-02-20T16:40:38.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:38.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247827 -lt 98784247827 2026-02-20T16:40:38.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514501 2026-02-20T16:40:38.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:38.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-326417514501 2026-02-20T16:40:38.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514501 2026-02-20T16:40:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 326417514501' 2026-02-20T16:40:38.708 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 326417514501 2026-02-20T16:40:38.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:38.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514501 -lt 326417514501 2026-02-20T16:40:38.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:38.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953490 2026-02-20T16:40:38.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:38.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:38.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953490 2026-02-20T16:40:38.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953490 2026-02-20T16:40:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953490' 2026-02-20T16:40:38.871 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953490 2026-02-20T16:40:38.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:39.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953490 -lt 137438953490 2026-02-20T16:40:39.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:39.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822673 2026-02-20T16:40:39.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:39.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:39.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822673 2026-02-20T16:40:39.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:39.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822673 2026-02-20T16:40:39.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822673' 2026-02-20T16:40:39.046 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 154618822673 2026-02-20T16:40:39.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:39.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822673 -lt 154618822673 2026-02-20T16:40:39.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:39.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659152 2026-02-20T16:40:39.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:39.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:39.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659152 2026-02-20T16:40:39.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:39.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659152 2026-02-20T16:40:39.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659152' 2026-02-20T16:40:39.220 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 176093659152 2026-02-20T16:40:39.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:39.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659152 -lt 176093659152 2026-02-20T16:40:39.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:39.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-352187318274 2026-02-20T16:40:39.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:39.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-352187318274 2026-02-20T16:40:39.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:39.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318274 2026-02-20T16:40:39.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 352187318274' 2026-02-20T16:40:39.388 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 352187318274 2026-02-20T16:40:39.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:39.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318274 -lt 352187318274 2026-02-20T16:40:39.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:39.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:39.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:39.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:39.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:39.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:39.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:39.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:39.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:39.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:40.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:40.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:40.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:40.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:40:40.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:40:40.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-20T16:40:40.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:40:40.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 188370"' 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 188370' 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 188372"' 2026-02-20T16:40:40.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 188372' 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 188370 188372' 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 188370 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/188373: /' 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:40:40.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/188375: /' 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:40.748 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:40.749 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: start osd.9 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:1activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:40:40.752 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: start osd.5 2026-02-20T16:40:40.753 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:42.250 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:40:42.250 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:40:42.250 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:42.250 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2026-02-20T16:40:40.760+0000 7f259bc52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2026-02-20T16:40:40.764+0000 7f259bc52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2026-02-20T16:40:40.764+0000 7f259bc52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:42.251 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: 0 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2026-02-20T16:40:41.228+0000 7f259bc52a40 -1 Falling back to public interface 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2026-02-20T16:40:41.920+0000 7f259bc52a40 -1 osd.9 83 log_to_monitors true 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: 1 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:42.252 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: osd.9 up in weight 1 up_from 87 up_thru 0 down_at 84 last_clean_interval [82,83) [v2:127.0.0.1:6842/2709700649,v1:127.0.0.1:6843/2709700649] [v2:127.0.0.1:6844/2709700649,v1:127.0.0.1:6845/2709700649] exists,up 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:42.253 INFO:tasks.workunit.client.0.trial013.stderr:188375: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:42.258 INFO:tasks.workunit.client.0.trial013.stderr:188375: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge88373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:40:42.258 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:40:42.258 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:42.258 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:42.258 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2026-02-20T16:40:40.760+0000 7fd1d947aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2026-02-20T16:40:40.764+0000 7fd1d947aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2026-02-20T16:40:40.764+0000 7fd1d947aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: 0 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:42.259 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2026-02-20T16:40:41.244+0000 7fd1d947aa40 -1 Falling back to public interface 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2026-02-20T16:40:41.964+0000 7fd1d947aa40 -1 osd.5 83 log_to_monitors true 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: 1 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: osd.5 up in weight 1 up_from 87 up_thru 87 down_at 84 last_clean_interval [76,83) [v2:127.0.0.1:6874/3410381383,v1:127.0.0.1:6875/3410381383] [v2:127.0.0.1:6876/3410381383,v1:127.0.0.1:6877/3410381383] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:42.260 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:42.261 INFO:tasks.workunit.client.0.trial013.stderr:188373: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:42.808 INFO:tasks.workunit.client.0.trial013.stderr:188373: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:40:42.808 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: 1 2026-02-20T16:40:42.809 INFO:tasks.workunit.client.0.trial013.stderr:188373: 2 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 3 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 4 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 5 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 6 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 7 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 8 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: 9' 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-20T16:40:42.810 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705688 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705688 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688' 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574871 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574871 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871' 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:42.811 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841549 2026-02-20T16:40:42.812 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841549 2026-02-20T16:40:42.812 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549' 2026-02-20T16:40:42.812 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.812 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:t_timeout_delays: echo true 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:42.813 INFO:tasks.workunit.client.0.trial013.stderr:188375: 1 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 2 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 3 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 4 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 5 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 6 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 7 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 8 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: 9' 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:42.814 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705689 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705689 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689' 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574872 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574872 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872' 2026-02-20T16:40:42.815 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:42.816 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:42.816 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841550 2026-02-20T16:40:42.816 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841550 2026-02-20T16:40:42.816 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550' 2026-02-20T16:40:42.816 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247830 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247830 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830' 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154755 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154755 2026-02-20T16:40:43.245 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830 5-373662154755' 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953493 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953493 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830 5-373662154755 6-137438953493' 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822675 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822675 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830 5-373662154755 6-137438953493 7-154618822675' 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659155 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659155 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830 5-373662154755 6-137438953493 7-154618822675 8-176093659155' 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154754 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154754 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-98784247830 5-373662154755 6-137438953493 7-154618822675 8-176093659155 9-373662154754' 2026-02-20T16:40:43.246 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:43.249 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/ceph flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:43.249 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247829 2026-02-20T16:40:43.249 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247829 2026-02-20T16:40:43.249 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829' 2026-02-20T16:40:43.249 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154754 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154754 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829 5-373662154754' 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953492 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953492 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829 5-373662154754 6-137438953492' 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.250 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822676 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822676 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829 5-373662154754 6-137438953492 7-154618822676' 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659156 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659156 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829 5-373662154754 6-137438953492 7-154618822676 8-176093659156' 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154755 2026-02-20T16:40:43.251 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154755 2026-02-20T16:40:43.252 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-98784247829 5-373662154754 6-137438953492 7-154618822676 8-176093659156 9-373662154755' 2026-02-20T16:40:43.252 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/ceptest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.891 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.0 seq 21474836505 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:44.892 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836505 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705689 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705689 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705689 2026-02-20T16:40:44.893 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705689' 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.1 seq 38654705689 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705689 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574872 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:44.894 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574872 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574872 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574872' 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.2 seq 55834574872 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:44.895 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574872 -lt 55834574872 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_statshtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-20T16:40:44.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.0 seq 21474836504 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836504 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705688 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:44.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705688 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705688 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705688' 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.1 seq 38654705688 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705688 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574871 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574871 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:44.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574871 2026-02-20T16:40:44.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574871' 2026-02-20T16:40:44.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.2 seq 55834574871 2026-02-20T16:40:44.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:44.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574872 -lt 55834574871 2026-02-20T16:40:45.383 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stat: for s in $seqs 2026-02-20T16:40:45.383 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841550 2026-02-20T16:40:45.383 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.383 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841550 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841550 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841550' 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.3 seq 283467841550 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841550 -lt 283467841550 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247830 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:45.384 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247830 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247830 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247830' 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.4 seq 98784247830 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247831 -lt 98784247830 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-373662154755 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-373662154755 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.385 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154755 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 373662154755' 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.5 seq 373662154755 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154755 -lt 373662154755 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953493 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.386 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:45.407 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-s: for s in $seqs 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841549 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841549 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841549 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841549' 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.3 seq 283467841549 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841550 -lt 283467841549 2026-02-20T16:40:45.408 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247829 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247829 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247829 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247829' 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.4 seq 98784247829 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247831 -lt 98784247829 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-373662154754 2026-02-20T16:40:45.409 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-373662154754 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154754 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 373662154754' 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.5 seq 373662154754 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154755 -lt 373662154754 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953492 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.410 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cephhelpers.sh:2268: flush_pg_stats: echo 6-137438953493 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953493 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953493' 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.6 seq 137438953493 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953493 -lt 137438953493 2026-02-20T16:40:45.873 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822675 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822675 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822675 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822675' 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.7 seq 154618822675 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822677 -lt 154618822675 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.874 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659155 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659155 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659155 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659155' 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.8 seq 176093659155 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659156 -lt 176093659155 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-373662154754 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.875 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:45.876 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-373662154754 2026-02-20T16:40:45.876 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.876 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154754 2026-02-20T16:40:45.906 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 37366215-helpers.sh:2268: flush_pg_stats: echo 6-137438953492 2026-02-20T16:40:45.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953492 2026-02-20T16:40:45.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953492' 2026-02-20T16:40:45.906 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.6 seq 137438953492 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953493 -lt 137438953492 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822676 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822676 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822676 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822676' 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.7 seq 154618822676 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:45.907 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822677 -lt 154618822676 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659156 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659156 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659156 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659156' 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.8 seq 176093659156 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659156 -lt 176093659156 2026-02-20T16:40:45.908 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-373662154755 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-373662154755 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:45.909 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154755 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 37366214754' 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: waiting osd.9 seq 373662154754 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154755 -lt 373662154754 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:46.583 INFO:tasks.workunit.client.0.trial013.stderr:188375: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:46.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:54755' 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: waiting osd.9 seq 373662154755 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154755 -lt 373662154755 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:46.634 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:40:46.635 INFO:tasks.workunit.client.0.trial013.stderr:188373: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:188373: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 188372 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:40:46.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:40:46.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:40:46.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:40:46.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:40:46.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:40:46.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:40:46.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:40:46.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:40:46.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:40:46.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:40:46.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:46.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:46.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:46.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:46.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:46.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:40:47.102 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:40:47.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:47.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:47.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:47.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:48.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:48.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:48.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:48.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:48.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:48.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:48.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:48.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:48.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:49.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:49.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:49.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:49.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:49.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:49.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:49.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:49.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:49.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:50.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:50.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:50.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:50.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:50.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:50.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:50.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:51.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:51.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:51.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:51.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:51.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:51.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:52.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:52.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:30.591950+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:40:54.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:40:54.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:40:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:40:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:40:54.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:40:54.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:40:54.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:30.591950+0000 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 191832"' 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 191832' 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:40:54.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 191833"' 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 191833' 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 191832 191833' 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 191832 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/191835: /' 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:54.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/191837: /' 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.767 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: _ 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: hinfo_key 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: snapset 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:54.768 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:54.770 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:40:54.771 INFO:tasks.workunit.client.0.trial013.stderr:191835: start osd.5 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:40:54.828 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:40:54.829 INFO:tasks.workunit.client.0.trial013.stderr:191837: _ 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: hinfo_key 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: snapset 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:40:54.830 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:40:54.831 INFO:tasks.workunit.client.0.trial013.stderr:191837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:40:54.832 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:40:54.833 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:40:54.833 INFO:tasks.workunit.client.0.trial013.stderr:191837: start osd.9 2026-02-20T16:40:56.258 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:40:56.258 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:40:56.258 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2026-02-20T16:40:54.784+0000 7f9eca9cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2026-02-20T16:40:54.784+0000 7f9eca9cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2026-02-20T16:40:54.784+0000 7f9eca9cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:56.259 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: 0 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2026-02-20T16:40:55.264+0000 7f9eca9cba40 -1 Falling back to public interface 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2026-02-20T16:40:55.956+0000 7f9eca9cba40 -1 osd.5 88 log_to_monitors true 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: 1 2026-02-20T16:40:56.260 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: osd.5 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [87,88) [v2:127.0.0.1:6842/203641157,v1:127.0.0.1:6843/203641157] [v2:127.0.0.1:6844/203641157,v1:127.0.0.1:6845/203641157] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:56.261 INFO:tasks.workunit.client.0.trial013.stderr:191835: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:56.299 INFO:tasks.workunit.client.0.trial013.stderr:191835: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1very_ops 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2026-02-20T16:40:54.840+0000 7fb88dd01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2026-02-20T16:40:54.840+0000 7fb88dd01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2026-02-20T16:40:54.844+0000 7fb88dd01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:40:56.300 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: 0 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2026-02-20T16:40:55.280+0000 7fb88dd01a40 -1 Falling back to public interface 2026-02-20T16:40:56.301 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2026-02-20T16:40:55.984+0000 7fb88dd01a40 -1 osd.9 88 log_to_monitors true 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: 1 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: osd.9 up in weight 1 up_from 92 up_thru 0 down_at 89 last_clean_interval [87,88) [v2:127.0.0.1:6874/3532684592,v1:127.0.0.1:6875/3532684592] [v2:127.0.0.1:6876/3532684592,v1:127.0.0.1:6877/3532684592] exists,up 1da8e7f8-9f5e-4312-9f00-c7d153459349 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:40:56.302 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:40:56.303 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:40:56.303 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:40:56.303 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:40:56.303 INFO:tasks.workunit.client.0.trial013.stderr:191837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s593: get_timeout_delays: echo true 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:56.821 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 1 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 2 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 3 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 4 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 5 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 6 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 7 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 8 2026-02-20T16:40:56.822 INFO:tasks.workunit.client.0.trial013.stderr:191835: 9' 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705692 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705692 2026-02-20T16:40:56.823 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692' 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574876 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574876 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876' 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841554 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841554 2026-02-20T16:40:56.824 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554' 2026-02-20T16:40:56.825 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.841 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shh:1593: get_timeout_delays: echo true 2026-02-20T16:40:56.841 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:40:56.841 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:40:56.841 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: 1 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: 2 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: 3 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: 4 2026-02-20T16:40:56.842 INFO:tasks.workunit.client.0.trial013.stderr:191837: 5 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: 6 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: 7 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: 8 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: 9' 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.843 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705693 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705693 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693' 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574877 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574877 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877' 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841555 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841555 2026-02-20T16:40:56.844 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555' 2026-02-20T16:40:56.845 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.265 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:57.265 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247834 2026-02-20T16:40:57.265 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247834 2026-02-20T16:40:57.265 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834' 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991234 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991234 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834 5-395136991234' 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953497 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953497 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834 5-395136991234 6-137438953497' 2026-02-20T16:40:57.266 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822680 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822680 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834 5-395136991234 6-137438953497 7-154618822680' 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659159 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659159 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834 5-395136991234 6-137438953497 7-154618822680 8-176093659159' 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991234 2026-02-20T16:40:57.267 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991234 2026-02-20T16:40:57.268 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574876 3-283467841554 4-98784247834 5-395136991234 6-137438953497 7-154618822680 8-176093659159 9-395136991234' 2026-02-20T16:40:57.268 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:57.280 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubun.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:40:57.280 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247835 2026-02-20T16:40:57.280 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247835 2026-02-20T16:40:57.280 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835' 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835 5-395136991235' 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953498 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953498 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835 5-395136991235 6-137438953498' 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.281 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822681 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822681 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835 5-395136991235 6-137438953498 7-154618822681' 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659160 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659160 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835 5-395136991235 6-137438953498 7-154618822681 8-176093659160' 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-20T16:40:57.282 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-55834574877 3-283467841555 4-98784247835 5-395136991235 6-137438953498 7-154618822681 8-176093659160 9-395136991235' 2026-02-20T16:40:57.283 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.925 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.0 seq 21474836510 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836510 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705693 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.926 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705693 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705693 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705693' 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.1 seq 38654705693 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705693 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574877 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:58.927 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574877 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574877 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574877' 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.2 seq 55834574877 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:58.928 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574877 -lt 55834574877 2026-02-20T16:40:58.942 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flustu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-20T16:40:58.942 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.942 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.0 seq 21474836509 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836509 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-20T16:40:58.943 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705692 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705692 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705692 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.1 seq 38654705692 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705692 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574876 2026-02-20T16:40:58.944 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574876 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574876 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574876' 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.2 seq 55834574876 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:40:58.945 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574877 -lt 55834574876 2026-02-20T16:40:59.414 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_ph_pg_stats: for s in $seqs 2026-02-20T16:40:59.415 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841555 2026-02-20T16:40:59.415 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.415 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:59.415 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841555 2026-02-20T16:40:59.415 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841555 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841555' 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.3 seq 283467841555 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841555 -lt 283467841555 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247835 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.416 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247835 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247835 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247835' 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.4 seq 98784247835 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247835 -lt 98784247835 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-395136991235 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-395136991235 2026-02-20T16:40:59.417 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991235 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 395136991235' 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.5 seq 395136991235 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953498 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.418 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:59.452 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standag_stats: for s in $seqs 2026-02-20T16:40:59.452 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841554 2026-02-20T16:40:59.452 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.452 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:40:59.452 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841554 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841554 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841554' 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.3 seq 283467841554 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:40:59.453 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841555 -lt 283467841554 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247834 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247834 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247834 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247834' 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.4 seq 98784247834 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247835 -lt 98784247834 2026-02-20T16:40:59.454 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-395136991234 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-395136991234 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991234 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 395136991234' 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.5 seq 395136991234 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991235 -lt 395136991234 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.455 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953497 2026-02-20T16:40:59.456 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.456 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:40:59.900 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalonlone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953498 2026-02-20T16:40:59.900 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.900 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953498 2026-02-20T16:40:59.900 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953498' 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.6 seq 137438953498 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953498 -lt 137438953498 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822681 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822681 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822681 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822681' 2026-02-20T16:40:59.901 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.7 seq 154618822681 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822682 -lt 154618822681 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659160 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659160 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659160 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659160' 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.8 seq 176093659160 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:59.902 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659161 -lt 176093659160 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991235 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991235 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.903 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991235 2026-02-20T16:40:59.953 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 see/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953497 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953497 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953497' 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.6 seq 137438953497 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953498 -lt 137438953497 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822680 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.954 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822680 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822680 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822680' 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.7 seq 154618822680 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822682 -lt 154618822680 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659159 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:40:59.955 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659159 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659159 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659159' 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.8 seq 176093659159 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659161 -lt 176093659159 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991234 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991234 2026-02-20T16:40:59.956 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:40:59.957 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991234 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 3q 395136991235' 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: waiting osd.9 seq 395136991235 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:00.636 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:00.637 INFO:tasks.workunit.client.0.trial013.stderr:191837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:00.638 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:00.638 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:00.638 INFO:tasks.workunit.client.0.trial013.stderr:191837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:00.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:00.642 INFO:tasks.workunit.client.0.trial013.stderr:95136991234' 2026-02-20T16:41:00.642 INFO:tasks.workunit.client.0.trial013.stderr:191835: waiting osd.9 seq 395136991234 2026-02-20T16:41:00.642 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:00.642 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991235 -lt 395136991234 2026-02-20T16:41:00.642 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:00.643 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:191835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:191835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:00.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:00.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:41:00.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 191833 2026-02-20T16:41:00.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:41:00.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:41:00.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:41:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:41:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:41:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:41:00.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:41:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-20T16:41:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:41:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:41:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 194705"' 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 194705' 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 194707"' 2026-02-20T16:41:00.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 194707' 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 194705 194707' 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 194705 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/194708: /' 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:41:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/194710: /' 2026-02-20T16:41:01.275 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.275 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:41:01.275 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:41:01.275 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:41:01.275 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:01.276 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:41:01.277 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:41:01.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:41:01.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:01.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:01.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:01.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:41:01.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:41:01.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:41:01.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:41:01.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: start osd.5 2026-02-20T16:41:01.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:1194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:41:01.324 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:41:01.325 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:41:01.326 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:41:01.327 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:01.327 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:01.328 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:01.329 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: start osd.3 2026-02-20T16:41:01.330 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:02.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:41:02.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:41:02.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2026-02-20T16:41:01.296+0000 7ff8af863a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2026-02-20T16:41:01.296+0000 7ff8af863a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2026-02-20T16:41:01.300+0000 7ff8af863a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:02.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: 0 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:02.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2026-02-20T16:41:01.851+0000 7ff8af863a40 -1 Falling back to public interface 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2026-02-20T16:41:02.567+0000 7ff8af863a40 -1 osd.5 93 log_to_monitors true 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: 1 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:02.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: osd.5 up in weight 1 up_from 98 up_thru 92 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6826/3277587794,v1:127.0.0.1:6827/3277587794] [v2:127.0.0.1:6828/3277587794,v1:127.0.0.1:6829/3277587794] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:41:02.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:41:02.809 INFO:tasks.workunit.client.0.trial013.stderr:194710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g94708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:41:02.809 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2026-02-20T16:41:01.340+0000 7f8d340e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2026-02-20T16:41:01.340+0000 7f8d340e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2026-02-20T16:41:01.344+0000 7f8d340e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:02.810 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: 0 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2026-02-20T16:41:01.875+0000 7f8d340e0a40 -1 Falling back to public interface 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2026-02-20T16:41:02.567+0000 7f8d340e0a40 -1 osd.3 93 log_to_monitors true 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:02.811 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: 1 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: osd.3 up in weight 1 up_from 98 up_thru 92 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6842/2206372257,v1:127.0.0.1:6843/2206372257] [v2:127.0.0.1:6844/2206372257,v1:127.0.0.1:6845/2206372257] exists,up 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:41:02.812 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:41:02.813 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:41:02.813 INFO:tasks.workunit.client.0.trial013.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:41:03.332 INFO:tasks.workunit.client.0.trial013.stderr:194708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:41:03.332 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: 1 2026-02-20T16:41:03.333 INFO:tasks.workunit.client.0.trial013.stderr:194710: 2 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 3 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 4 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 5 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 6 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 7 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 8 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: 9' 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-20T16:41:03.334 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705696 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705696 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696' 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574879 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574879 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879' 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.335 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:41:03.336 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795010 2026-02-20T16:41:03.336 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795010 2026-02-20T16:41:03.336 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010' 2026-02-20T16:41:03.336 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.356 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:et_timeout_delays: echo true 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:41:03.357 INFO:tasks.workunit.client.0.trial013.stderr:194708: 1 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 2 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 3 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 4 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 5 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 6 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 7 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 8 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: 9' 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:41:03.358 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705697 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705697 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697' 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574880 2026-02-20T16:41:03.359 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574880 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880' 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795011 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795011 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011' 2026-02-20T16:41:03.360 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247838 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247838 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838' 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795010 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795010 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838 5-420906795010' 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953500 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953500 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838 5-420906795010 6-137438953500' 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822683 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822683 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838 5-420906795010 6-137438953500 7-154618822683' 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659163 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659163 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838 5-420906795010 6-137438953500 7-154618822683 8-176093659163' 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991237 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991237 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-420906795010 4-98784247838 5-420906795010 6-137438953500 7-154618822683 8-176093659163 9-395136991237' 2026-02-20T16:41:03.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cep flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247839 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247839 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839' 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.789 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795011 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795011 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839 5-420906795011' 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953501 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953501 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839 5-420906795011 6-137438953501' 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822684 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822684 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839 5-420906795011 6-137438953501 7-154618822684' 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:41:03.790 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659164 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659164 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839 5-420906795011 6-137438953501 7-154618822684 8-176093659164' 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991238 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991238 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-420906795011 4-98784247839 5-420906795011 6-137438953501 7-154618822684 8-176093659164 9-395136991238' 2026-02-20T16:41:03.791 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.278 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-20T16:41:04.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:41:04.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-20T16:41:04.278 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.0 seq 21474836512 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836512 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705696 2026-02-20T16:41:04.279 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705696 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705696 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705696' 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.1 seq 38654705696 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705696 2026-02-20T16:41:04.280 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574879 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574879 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574879 2026-02-20T16:41:04.281 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574879' 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.2 seq 55834574879 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574880 -lt 55834574879 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-420906795010 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:41:04.282 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-420906795010 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194htest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-20T16:41:04.293 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.0 seq 21474836513 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836513 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705697 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705697 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705697 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.1 seq 38654705697 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:41:04.294 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705697 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574880 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574880 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574880 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574880' 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.2 seq 55834574880 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574880 -lt 55834574880 2026-02-20T16:41:04.295 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.296 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-420906795011 2026-02-20T16:41:04.296 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.296 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:41:04.296 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-420906795011 2026-02-20T16:41:04.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795010 2026-02-20T16:41:04.773 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 420906795010' 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.3 seq 420906795010 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795011 -lt 420906795010 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247838 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247838 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247838 2026-02-20T16:41:04.774 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247838' 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.4 seq 98784247838 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247839 -lt 98784247838 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-420906795010 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-420906795010 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795010 2026-02-20T16:41:04.775 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 420906795010' 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.5 seq 420906795010 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795010 -lt 420906795010 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953500 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953500 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:04.776 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953500 2026-02-20T16:41:04.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953500' 2026-02-20T16:41:04.777 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.6 seq 137438953500 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubun708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795011 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 420906795011' 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.3 seq 420906795011 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:41:05.948 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795011 -lt 420906795011 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247839 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247839 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247839 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247839' 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.4 seq 98784247839 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:41:05.949 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247839 -lt 98784247839 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-420906795011 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-420906795011 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795011 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 420906795011' 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.5 seq 420906795011 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795010 -lt 420906795011 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:41:05.950 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795011 -lt 420906795011 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953501 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:05.951 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:41:06.432 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cetu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:41:06.432 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953499 -lt 137438953500 2026-02-20T16:41:06.432 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:41:06.432 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:41:06.432 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953501 -lt 137438953500 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822683 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822683 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822683 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822683' 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.7 seq 154618822683 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:41:06.433 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822685 -lt 154618822683 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659163 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659163 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659163 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659163' 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.8 seq 176093659163 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659164 -lt 176093659163 2026-02-20T16:41:06.434 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991237 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991237 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.435 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991237 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 39513ph-helpers.sh:2268: flush_pg_stats: echo 6-137438953501 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953501 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953501' 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.6 seq 137438953501 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953501 -lt 137438953501 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822684 2026-02-20T16:41:06.436 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822684 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822684 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822684' 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.7 seq 154618822684 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822685 -lt 154618822684 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659164 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:41:06.437 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659164 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659164 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659164' 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.8 seq 176093659164 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659164 -lt 176093659164 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991238 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991238 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:06.438 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991238 2026-02-20T16:41:07.132 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 395136991238' 2026-02-20T16:41:07.132 INFO:tasks.workunit.client.0.trial013.stderr:194708: waiting osd.9 seq 395136991238 2026-02-20T16:41:07.132 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:07.132 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991238 -lt 395136991238 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:07.133 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:194708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:41:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 194707 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:6991237' 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:194710: waiting osd.9 seq 395136991237 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991238 -lt 395136991237 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:07.159 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:07.160 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:194710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:194710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:41:07.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:41:07.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:41:07.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:07.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:07.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:07.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:41:07.628 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:41:07.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:07.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:07.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:07.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:07.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:07.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:41:08.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:08.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:08.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:08.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:41:09.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:41:09.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:09.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:09.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:09.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:09.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:09.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:10.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:10.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:41:11.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:41:11.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:11.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:11.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:11.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:11.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:11.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:11.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:11.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:41:12.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:12.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:12.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:40:47.585070+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:12.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:41:13.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:41:13.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:41:13.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:41:13.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:41:13.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:41:13.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:41:13.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:41:13.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:41:08.481809+0000 '>' 2026-02-20T16:40:47.585070+0000 2026-02-20T16:41:13.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:41:13.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 198142"' 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 198142' 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 198143"' 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 198143' 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:41:13.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 198142 198143' 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 198142 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/198145: /' 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:41:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/198147: /' 2026-02-20T16:41:14.143 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:41:14.144 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:14.145 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:41:14.146 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: _ 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: hinfo_key 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: snapset 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:41:14.147 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:41:14.148 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:14.149 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:41:14.150 INFO:tasks.workunit.client.0.trial013.stderr:198145: start osd.3 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:41:14.164 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.165 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: _ 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: hinfo_key 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: snapset 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:41:14.166 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:41:14.167 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:14.168 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:41:14.169 INFO:tasks.workunit.client.0.trial013.stderr:198147: start osd.5 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recolog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:41:15.634 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2026-02-20T16:41:14.159+0000 7f9b9c879a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2026-02-20T16:41:14.159+0000 7f9b9c879a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2026-02-20T16:41:14.163+0000 7f9b9c879a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 0 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2026-02-20T16:41:14.659+0000 7f9b9c879a40 -1 Falling back to public interface 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2026-02-20T16:41:15.355+0000 7f9b9c879a40 -1 osd.3 99 log_to_monitors true 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: 1 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: osd.3 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [98,99) [v2:127.0.0.1:6826/2081233220,v1:127.0.0.1:6827/2081233220] [v2:127.0.0.1:6828/2081233220,v1:127.0.0.1:6829/2081233220] exists,up 3403e1c9-3ccd-4ce7-a997-4baa92e9c69a 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:41:15.635 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_very_ops 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2026-02-20T16:41:14.179+0000 7fa5f0a6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.649 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2026-02-20T16:41:14.179+0000 7fa5f0a6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2026-02-20T16:41:14.179+0000 7fa5f0a6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: 0 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:15.650 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2026-02-20T16:41:14.679+0000 7fa5f0a6ca40 -1 Falling back to public interface 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2026-02-20T16:41:15.359+0000 7fa5f0a6ca40 -1 osd.5 99 log_to_monitors true 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: 1 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: osd.5 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [98,99) [v2:127.0.0.1:6842/3492438334,v1:127.0.0.1:6843/3492438334] [v2:127.0.0.1:6844/3492438334,v1:127.0.0.1:6845/3492438334] exists,up 69574da6-9125-48f9-bca2-c80d60bd439c 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:15.651 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:41:15.652 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:41:15.652 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:41:15.652 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:41:15.652 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:41:15.652 INFO:tasks.workunit.client.0.trial013.stderr:198147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:41:16.199 INFO:tasks.workunit.client.0.trial013.stderr:198147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpedelays 90 .1 2026-02-20T16:41:16.199 INFO:tasks.workunit.client.0.trial013.stderr:198145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:41:16.199 INFO:tasks.workunit.client.0.trial013.stderr:198145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 1 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 2 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 3 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 4 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 5 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 6 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 7 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 8 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: 9' 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836517 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836517 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517' 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700' 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574884 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574884 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884' 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631490 2026-02-20T16:41:16.200 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631490 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-558rs.sh:1593: get_timeout_delays: echo true 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: 1 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: 2 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: 3 2026-02-20T16:41:16.204 INFO:tasks.workunit.client.0.trial013.stderr:198147: 4 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: 5 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: 6 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: 7 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: 8 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: 9' 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705701 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705701 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701' 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574885 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574885 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885' 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631491 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631491 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491' 2026-02-20T16:41:16.205 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.638 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:41:16.638 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247843 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247843 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843' 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631491 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631491 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843 5-442381631491' 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953505 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953505 2026-02-20T16:41:16.639 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843 5-442381631491 6-137438953505' 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822688 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822688 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843 5-442381631491 6-137438953505 7-154618822688' 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659167 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659167 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843 5-442381631491 6-137438953505 7-154618822688 8-176093659167' 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:41:16.640 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991242 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991242 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-442381631491 4-98784247843 5-442381631491 6-137438953505 7-154618822688 8-176093659167 9-395136991242' 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198147: //ho34574884 3-442381631490' 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.641 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247842 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247842 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842' 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631490 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631490 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842 5-442381631490' 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953506 2026-02-20T16:41:16.642 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953506 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842 5-442381631490 6-137438953506' 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822689 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822689 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842 5-442381631490 6-137438953506 7-154618822689' 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659168 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659168 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842 5-442381631490 6-137438953506 7-154618822689 8-176093659168' 2026-02-20T16:41:16.643 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:16.644 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:41:16.644 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991243 2026-02-20T16:41:16.644 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991243 2026-02-20T16:41:18.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-442381631490 4-98784247842 5-442381631490 6-137438953506 7-154618822689 8-176093659168 9-395136991243' 2026-02-20T16:41:18.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836517 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836517 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836517 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2026-02-20T16:41:18.297 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.0 seq 21474836517 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836517 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836517 2026-02-20T16:41:18.298 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-20T16:41:18.299 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.1 seq 38654705700 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705700 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574884 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:41:18.300 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574884 2026-02-20T16:41:18.301 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.301 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574884 2026-02-20T16:41:18.301 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574884' 2026-02-20T16:41:18.301 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.2 seq 55834574884 2026-02-20T16:41:18.301 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:41:18.317 INFO:tasks.workunit.client.0.trial013.stderr:19814me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-20T16:41:18.317 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.317 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:41:18.317 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.0 seq 21474836518 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-20T16:41:18.318 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836518 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705701 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705701 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705701 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2026-02-20T16:41:18.319 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.1 seq 38654705701 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705701 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574885 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574885 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574885 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574885' 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.2 seq 55834574885 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:41:18.320 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574885 -lt 55834574885 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: 5: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574885 -lt 55834574884 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-442381631490 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:41:18.787 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-442381631490 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631490 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 442381631490' 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.3 seq 442381631490 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631490 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247842 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247842 2026-02-20T16:41:18.788 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247842 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247842' 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.4 seq 98784247842 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247843 -lt 98784247842 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-442381631490 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-442381631490 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.789 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631490 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 442381631490' 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.5 seq 442381631490 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631490 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.790 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953506 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cflush_pg_stats: for s in $seqs 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-442381631491 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-442381631491 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631491 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 442381631491' 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.3 seq 442381631491 2026-02-20T16:41:18.822 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631491 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247843 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247843 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247843 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247843' 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.4 seq 98784247843 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:41:18.823 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247843 -lt 98784247843 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-442381631491 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-442381631491 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631491 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 442381631491' 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.5 seq 442381631491 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631491 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:18.824 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953505 2026-02-20T16:41:18.825 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:18.825 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:41:19.293 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/steph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.293 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953506 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953506 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953506' 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.6 seq 137438953506 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953506 -lt 137438953506 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822689 2026-02-20T16:41:19.294 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822689 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822689 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822689' 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.7 seq 154618822689 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822689 -lt 154618822689 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659168 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:41:19.295 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659168 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659168 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659168' 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.8 seq 176093659168 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659168 -lt 176093659168 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991243 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991243 2026-02-20T16:41:19.296 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cepandalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953505 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953505 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953505' 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.6 seq 137438953505 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:41:19.309 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953506 -lt 137438953505 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822688 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-154618822688 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822688 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 154618822688' 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.7 seq 154618822688 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822689 -lt 154618822688 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.310 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-176093659167 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-176093659167 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659167 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 176093659167' 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.8 seq 176093659167 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659168 -lt 176093659167 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991242 2026-02-20T16:41:19.311 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:41:19.312 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:41:19.312 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-395136991242 2026-02-20T16:41:19.312 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:41:19.312 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991242 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 395136991242' 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: waiting osd.9 seq 395136991242 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991243 -lt 395136991242 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:20.010 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:20.011 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:20.011 INFO:tasks.workunit.client.0.trial013.stderr:198147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:20.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:htest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=395136991243 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 395136991243' 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: waiting osd.9 seq 395136991243 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 395136991243 -lt 395136991243 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:41:20.022 INFO:tasks.workunit.client.0.trial013.stderr:198145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 198143 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:41:20.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:41:20.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:20.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:20.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:20.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:20.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:20.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:41:20.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:41:20.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:41:20.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:41:20.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:41:20.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:41:20.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:41:20.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:41:20.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:41:20.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:41:20.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:41:20.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:41:20.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:41:20.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:41:20.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:41:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:41:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:41:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:41:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:41:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:41:20.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:41:20.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:41:20.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:41:20.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:41:20.209 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:41:20.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:41:20.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:41:20.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:41:20.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:41:20.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:41:20.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:41:20.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:41:20.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:41:20.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:41:20.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:41:20.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:41:20.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:41:20.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:41:20.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:41:20.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:41:20.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:41:20.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:41:20.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:905: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=true 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:41:20.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:41:20.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:41:20.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:20.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:20.245 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:20.245 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.245 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:20.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:41:20.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:41:20.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:41:20.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:41:20.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:41:20.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:41:20.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:41:20.275 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:41:20.275 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:41:20.275 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:41:20.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:41:20.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:41:20.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:41:20.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:41:20.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:41:20.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:41:20.332 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:41:20.332 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:41:20.332 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:41:20.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:41:20.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:41:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:41:20.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:41:20.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:20.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:20.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:20.485 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.485 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:41:20.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:41:20.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:20.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:20.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:41:20.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b8594f2e-1c8f-4396-a7c8-eb1056880eb1 2026-02-20T16:41:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b8594f2e-1c8f-4396-a7c8-eb1056880eb1' 2026-02-20T16:41:20.510 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 b8594f2e-1c8f-4396-a7c8-eb1056880eb1 2026-02-20T16:41:20.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:20.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwjphpUzQMHxAAI7h5J3BXzafM9moL1f7kHQ== 2026-02-20T16:41:20.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwjphpUzQMHxAAI7h5J3BXzafM9moL1f7kHQ=="}' 2026-02-20T16:41:20.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b8594f2e-1c8f-4396-a7c8-eb1056880eb1 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:41:20.610 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:20.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:41:20.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwjphpUzQMHxAAI7h5J3BXzafM9moL1f7kHQ== --osd-uuid b8594f2e-1c8f-4396-a7c8-eb1056880eb1 2026-02-20T16:41:20.634 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:20.631+0000 7f189d871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:20.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:20.635+0000 7f189d871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:20.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:20.635+0000 7f189d871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:20.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:20.635+0000 7f189d871a40 -1 bdev(0x55bba0f9ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:20.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:20.635+0000 7f189d871a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:41:21.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:41:21.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:21.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:41:21.392 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:41:21.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:21.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:41:21.496 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:41:21.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:21.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:21.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:21.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:21.521 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:21.519+0000 7fe0cea49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:21.523 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:21.523+0000 7fe0cea49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:21.525 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:21.523+0000 7fe0cea49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:21.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:21.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:21.594 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:21.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:21.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:41:21.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:21.945 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:21.943+0000 7fe0cea49a40 -1 Falling back to public interface 2026-02-20T16:41:22.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:22.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:22.687 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:22.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:22.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:22.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:41:22.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:22.935 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:22.931+0000 7fe0cea49a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:41:23.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:23.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:23.779 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:23.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:23.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:23.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:41:23.939 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4059235484,v1:127.0.0.1:6801/4059235484] [v2:127.0.0.1:6802/4059235484,v1:127.0.0.1:6803/4059235484] exists,up b8594f2e-1c8f-4396-a7c8-eb1056880eb1 2026-02-20T16:41:23.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:23.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:23.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:23.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:23.944 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:23.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:41:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:23.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d811052-da60-45f2-bd42-aaa49b5edc1d 2026-02-20T16:41:23.947 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 5d811052-da60-45f2-bd42-aaa49b5edc1d 2026-02-20T16:41:23.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5d811052-da60-45f2-bd42-aaa49b5edc1d' 2026-02-20T16:41:23.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:23.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzjphpcK8AORAAbNseFXQ2hyGRcUEjf56pfw== 2026-02-20T16:41:23.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzjphpcK8AORAAbNseFXQ2hyGRcUEjf56pfw=="}' 2026-02-20T16:41:23.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d811052-da60-45f2-bd42-aaa49b5edc1d -i td/osd-scrub-repair/1/new.json 2026-02-20T16:41:24.121 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:24.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:41:24.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzjphpcK8AORAAbNseFXQ2hyGRcUEjf56pfw== --osd-uuid 5d811052-da60-45f2-bd42-aaa49b5edc1d 2026-02-20T16:41:24.150 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:24.147+0000 7f08bda9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:24.151 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:24.147+0000 7f08bda9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:24.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:24.151+0000 7f08bda9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:24.152 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:24.151+0000 7f08bda9aa40 -1 bdev(0x55aacd1c0c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:24.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:24.151+0000 7f08bda9aa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:41:24.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:41:24.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:24.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:41:24.824 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:41:24.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:25.022 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:41:25.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:41:25.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:25.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:25.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:25.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:25.046 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:25.043+0000 7f0d6b4c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:25.048 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:25.047+0000 7f0d6b4c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:25.049 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:25.047+0000 7f0d6b4c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:25.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:25.184 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:41:25.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:25.469 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:25.467+0000 7f0d6b4c1a40 -1 Falling back to public interface 2026-02-20T16:41:26.029 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:26.027+0000 7f0d6b4c1a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:41:26.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:26.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:26.352 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:26.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:26.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:26.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:41:26.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:41:27.677 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2221427555,v1:127.0.0.1:6811/2221427555] [v2:127.0.0.1:6812/2221427555,v1:127.0.0.1:6813/2221427555] exists,up 5d811052-da60-45f2-bd42-aaa49b5edc1d 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:27.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:27.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:27.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:41:27.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b0bee795-4897-473f-a301-cdf76a9e1d4c 2026-02-20T16:41:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b0bee795-4897-473f-a301-cdf76a9e1d4c' 2026-02-20T16:41:27.686 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 b0bee795-4897-473f-a301-cdf76a9e1d4c 2026-02-20T16:41:27.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:27.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3jphp4N5mKRAA2WcyimACcDu5+A6TUjcf5A== 2026-02-20T16:41:27.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3jphp4N5mKRAA2WcyimACcDu5+A6TUjcf5A=="}' 2026-02-20T16:41:27.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b0bee795-4897-473f-a301-cdf76a9e1d4c -i td/osd-scrub-repair/2/new.json 2026-02-20T16:41:27.859 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:27.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:41:27.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3jphp4N5mKRAA2WcyimACcDu5+A6TUjcf5A== --osd-uuid b0bee795-4897-473f-a301-cdf76a9e1d4c 2026-02-20T16:41:27.886 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:27.883+0000 7f9ae9f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:27.887 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:27.887+0000 7f9ae9f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:27.888 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:27.887+0000 7f9ae9f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:27.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:27.887+0000 7f9ae9f3fa40 -1 bdev(0x55e7963aec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:27.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:27.887+0000 7f9ae9f3fa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:41:28.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:41:28.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:28.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:41:28.548 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:41:28.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:28.756 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:41:28.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:41:28.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:28.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:28.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:28.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:28.779 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:28.775+0000 7ff3eb986a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:28.781 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:28.779+0000 7ff3eb986a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:28.782 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:28.779+0000 7ff3eb986a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:28.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:28.915 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:28.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:28.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:41:29.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:29.165 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:29.163+0000 7ff3eb986a40 -1 Falling back to public interface 2026-02-20T16:41:29.753 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:29.751+0000 7ff3eb986a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:41:30.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:30.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:30.079 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:30.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:30.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:30.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:41:30.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:31.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:31.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:31.239 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:31.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:31.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:31.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:41:31.400 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/964247961,v1:127.0.0.1:6819/964247961] [v2:127.0.0.1:6820/964247961,v1:127.0.0.1:6821/964247961] exists,up b0bee795-4897-473f-a301-cdf76a9e1d4c 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:31.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:31.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:31.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:31.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:41:31.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:31.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:41:31.408 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:41:31.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 f51cf9ba-a841-4d1d-b4c3-55aa0e231d25' 2026-02-20T16:41:31.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:31.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7jphpufbnGBAAnFirYDxC3J1iIvAskIA/1Q== 2026-02-20T16:41:31.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7jphpufbnGBAAnFirYDxC3J1iIvAskIA/1Q=="}' 2026-02-20T16:41:31.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 -i td/osd-scrub-repair/3/new.json 2026-02-20T16:41:31.573 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T16:41:31.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T16:41:31.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7jphpufbnGBAAnFirYDxC3J1iIvAskIA/1Q== --osd-uuid f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:41:31.600 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:31.599+0000 7f9acab1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:31.602 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:31.599+0000 7f9acab1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:31.603 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:31.599+0000 7f9acab1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:31.604 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:31.603+0000 7f9acab1fa40 -1 bdev(0x55fe31e92c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:31.604 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:31.603+0000 7f9acab1fa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T16:41:32.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T16:41:32.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:32.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T16:41:32.271 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T16:41:32.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:32.469 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T16:41:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T16:41:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:32.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:32.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:32.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:32.493 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:32.491+0000 7f2f6f789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:32.495 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:32.491+0000 7f2f6f789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:32.496 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:32.495+0000 7f2f6f789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:32.629 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:32.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:32.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:32.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:32.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:32.913 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:32.911+0000 7f2f6f789a40 -1 Falling back to public interface 2026-02-20T16:41:33.465 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:33.463+0000 7f2f6f789a40 -1 osd.3 0 log_to_monitors true 2026-02-20T16:41:33.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:33.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:33.793 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:33.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:33.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:33.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:33.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:34.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:41:35.109 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3635745178,v1:127.0.0.1:6827/3635745178] [v2:127.0.0.1:6828/3635745178,v1:127.0.0.1:6829/3635745178] exists,up f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-20T16:41:35.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:35.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:35.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:35.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:35.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:35.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:35.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-20T16:41:35.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:35.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2030739-fcf3-4568-b38d-aa8346930cc9 2026-02-20T16:41:35.114 INFO:tasks.workunit.client.0.trial013.stdout:add osd4 f2030739-fcf3-4568-b38d-aa8346930cc9 2026-02-20T16:41:35.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 f2030739-fcf3-4568-b38d-aa8346930cc9' 2026-02-20T16:41:35.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:35.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/jphpUoyFBxAADcGf6IsgEln9nCw5x6nAOQ== 2026-02-20T16:41:35.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/jphpUoyFBxAADcGf6IsgEln9nCw5x6nAOQ=="}' 2026-02-20T16:41:35.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2030739-fcf3-4568-b38d-aa8346930cc9 -i td/osd-scrub-repair/4/new.json 2026-02-20T16:41:35.290 INFO:tasks.workunit.client.0.trial013.stdout:4 2026-02-20T16:41:35.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-20T16:41:35.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/jphpUoyFBxAADcGf6IsgEln9nCw5x6nAOQ== --osd-uuid f2030739-fcf3-4568-b38d-aa8346930cc9 2026-02-20T16:41:35.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:35.315+0000 7f2160059a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:35.318 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:35.315+0000 7f2160059a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:35.319 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:35.315+0000 7f2160059a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:35.319 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:35.319+0000 7f2160059a40 -1 bdev(0x55c53b994c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:35.319 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:35.319+0000 7f2160059a40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-20T16:41:36.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-20T16:41:36.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-20T16:41:36.015 INFO:tasks.workunit.client.0.trial013.stdout:adding osd4 key to auth repository 2026-02-20T16:41:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:36.227 INFO:tasks.workunit.client.0.trial013.stdout:start osd.4 2026-02-20T16:41:36.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-20T16:41:36.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:36.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:36.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:36.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:36.250 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:36.247+0000 7f984650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:36.252 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:36.251+0000 7f984650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:36.253 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:36.251+0000 7f984650ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:36.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:36.392 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:36.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:36.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:36.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:41:36.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:36.681 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:36.679+0000 7f984650ba40 -1 Falling back to public interface 2026-02-20T16:41:37.274 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:37.271+0000 7f984650ba40 -1 osd.4 0 log_to_monitors true 2026-02-20T16:41:37.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:37.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:37.549 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:37.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:37.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:37.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:41:37.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:38.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:38.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:38.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:38.713 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-20T16:41:38.872 INFO:tasks.workunit.client.0.trial013.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/986670768,v1:127.0.0.1:6835/986670768] [v2:127.0.0.1:6836/986670768,v1:127.0.0.1:6837/986670768] exists,up f2030739-fcf3-4568-b38d-aa8346930cc9 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:38.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:38.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:38.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:38.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:41:38.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:38.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:41:38.880 INFO:tasks.workunit.client.0.trial013.stdout:add osd5 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:41:38.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 59df7655-1e17-4ef4-86c6-42ed9af0be0a' 2026-02-20T16:41:38.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:38.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCjphpTkgBNRAABmYvFDwvJmFIj5lX/0QrXw== 2026-02-20T16:41:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCjphpTkgBNRAABmYvFDwvJmFIj5lX/0QrXw=="}' 2026-02-20T16:41:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 59df7655-1e17-4ef4-86c6-42ed9af0be0a -i td/osd-scrub-repair/5/new.json 2026-02-20T16:41:39.051 INFO:tasks.workunit.client.0.trial013.stdout:5 2026-02-20T16:41:39.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-20T16:41:39.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCjphpTkgBNRAABmYvFDwvJmFIj5lX/0QrXw== --osd-uuid 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:41:39.077 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.075+0000 7f9b3a4fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:39.079 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.075+0000 7f9b3a4fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:39.080 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.079+0000 7f9b3a4fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:39.080 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.079+0000 7f9b3a4fea40 -1 bdev(0x5560118ccc00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:39.081 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.079+0000 7f9b3a4fea40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-20T16:41:39.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-20T16:41:39.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:39.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-20T16:41:39.703 INFO:tasks.workunit.client.0.trial013.stdout:adding osd5 key to auth repository 2026-02-20T16:41:39.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stdout:start osd.5 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:39.919 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.919+0000 7f77f6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:39.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.919+0000 7f77f6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:39.922 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:39.919+0000 7f77f6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:40.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:40.056 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:40.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:40.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:40.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:40.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:40.369 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:40.367+0000 7f77f6abaa40 -1 Falling back to public interface 2026-02-20T16:41:41.031 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:41.027+0000 7f77f6abaa40 -1 osd.5 0 log_to_monitors true 2026-02-20T16:41:41.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:41.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:41.213 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:41.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:41.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:41.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:41.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:42.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:41:42.536 INFO:tasks.workunit.client.0.trial013.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2602210082,v1:127.0.0.1:6843/2602210082] [v2:127.0.0.1:6844/2602210082,v1:127.0.0.1:6845/2602210082] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:42.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:42.539 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:42.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-20T16:41:42.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:42.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cf6dc531-2c02-45f9-ba02-4f7687d45557 2026-02-20T16:41:42.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 cf6dc531-2c02-45f9-ba02-4f7687d45557' 2026-02-20T16:41:42.542 INFO:tasks.workunit.client.0.trial013.stdout:add osd6 cf6dc531-2c02-45f9-ba02-4f7687d45557 2026-02-20T16:41:42.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:42.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDGjphpTJL/IBAAch7MzMQOzQGBeE7IFeRqKg== 2026-02-20T16:41:42.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDGjphpTJL/IBAAch7MzMQOzQGBeE7IFeRqKg=="}' 2026-02-20T16:41:42.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cf6dc531-2c02-45f9-ba02-4f7687d45557 -i td/osd-scrub-repair/6/new.json 2026-02-20T16:41:42.718 INFO:tasks.workunit.client.0.trial013.stdout:6 2026-02-20T16:41:42.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-20T16:41:42.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGjphpTJL/IBAAch7MzMQOzQGBeE7IFeRqKg== --osd-uuid cf6dc531-2c02-45f9-ba02-4f7687d45557 2026-02-20T16:41:42.754 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:42.751+0000 7f610274aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:42.755 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:42.755+0000 7f610274aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:42.756 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:42.755+0000 7f610274aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:42.757 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:42.755+0000 7f610274aa40 -1 bdev(0x555680d18c00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:42.757 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:42.755+0000 7f610274aa40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-20T16:41:43.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-20T16:41:43.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:43.468 INFO:tasks.workunit.client.0.trial013.stdout:adding osd6 key to auth repository 2026-02-20T16:41:43.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-20T16:41:43.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:43.668 INFO:tasks.workunit.client.0.trial013.stdout:start osd.6 2026-02-20T16:41:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-20T16:41:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:43.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:43.692 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:43.691+0000 7fa488882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:43.693 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:43.691+0000 7fa488882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:43.694 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:43.691+0000 7fa488882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:43.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-20T16:41:43.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:43.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-20T16:41:43.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:43.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:43.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:43.831 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:43.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:43.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:43.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:41:43.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:44.101 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:44.099+0000 7fa488882a40 -1 Falling back to public interface 2026-02-20T16:41:44.729 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:44.727+0000 7fa488882a40 -1 osd.6 0 log_to_monitors true 2026-02-20T16:41:44.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:44.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:44.992 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:44.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:44.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:44.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:41:45.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:46.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:46.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:46.152 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:46.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:46.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:46.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-20T16:41:46.310 INFO:tasks.workunit.client.0.trial013.stdout:osd.6 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1689368833,v1:127.0.0.1:6851/1689368833] [v2:127.0.0.1:6852/1689368833,v1:127.0.0.1:6853/1689368833] exists,up cf6dc531-2c02-45f9-ba02-4f7687d45557 2026-02-20T16:41:46.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-20T16:41:46.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:46.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:46.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:46.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:46.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:46.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:46.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-20T16:41:46.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:46.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=34aa9f2a-8042-41af-bfc9-13c7f162189b 2026-02-20T16:41:46.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 34aa9f2a-8042-41af-bfc9-13c7f162189b' 2026-02-20T16:41:46.318 INFO:tasks.workunit.client.0.trial013.stdout:add osd7 34aa9f2a-8042-41af-bfc9-13c7f162189b 2026-02-20T16:41:46.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:46.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKjphpaj6SExAAiYrbbL7V+TGoHmuyJAZ6Sw== 2026-02-20T16:41:46.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKjphpaj6SExAAiYrbbL7V+TGoHmuyJAZ6Sw=="}' 2026-02-20T16:41:46.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 34aa9f2a-8042-41af-bfc9-13c7f162189b -i td/osd-scrub-repair/7/new.json 2026-02-20T16:41:46.489 INFO:tasks.workunit.client.0.trial013.stdout:7 2026-02-20T16:41:46.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-20T16:41:46.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKjphpaj6SExAAiYrbbL7V+TGoHmuyJAZ6Sw== --osd-uuid 34aa9f2a-8042-41af-bfc9-13c7f162189b 2026-02-20T16:41:46.516 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:46.515+0000 7fefed596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:46.518 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:46.515+0000 7fefed596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:46.518 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:46.515+0000 7fefed596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:46.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:46.515+0000 7fefed596a40 -1 bdev(0x55a2f080cc00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:46.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:46.515+0000 7fefed596a40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-20T16:41:47.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-20T16:41:47.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:47.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-20T16:41:47.179 INFO:tasks.workunit.client.0.trial013.stdout:adding osd7 key to auth repository 2026-02-20T16:41:47.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:47.381 INFO:tasks.workunit.client.0.trial013.stdout:start osd.7 2026-02-20T16:41:47.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-20T16:41:47.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:47.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:47.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:47.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:47.405 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:47.403+0000 7f69fe1e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:47.407 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:47.403+0000 7f69fe1e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:47.408 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:47.407+0000 7f69fe1e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:47.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-20T16:41:47.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:47.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-20T16:41:47.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:47.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:47.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:47.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:47.543 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:47.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:47.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:41:47.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:47.817 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:47.815+0000 7f69fe1e9a40 -1 Falling back to public interface 2026-02-20T16:41:48.370 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:48.367+0000 7f69fe1e9a40 -1 osd.7 0 log_to_monitors true 2026-02-20T16:41:48.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:48.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:48.700 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:48.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:48.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:48.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:41:48.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:49.864 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:49.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:49.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:49.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-20T16:41:50.020 INFO:tasks.workunit.client.0.trial013.stdout:osd.7 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3601212317,v1:127.0.0.1:6859/3601212317] [v2:127.0.0.1:6860/3601212317,v1:127.0.0.1:6861/3601212317] exists,up 34aa9f2a-8042-41af-bfc9-13c7f162189b 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:50.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:50.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-20T16:41:50.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:50.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=59a76eec-2434-406f-bd98-2c41c3dcfe82 2026-02-20T16:41:50.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 59a76eec-2434-406f-bd98-2c41c3dcfe82' 2026-02-20T16:41:50.026 INFO:tasks.workunit.client.0.trial013.stdout:add osd8 59a76eec-2434-406f-bd98-2c41c3dcfe82 2026-02-20T16:41:50.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:50.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOjphpUGZNAhAAN9XRfGqmQq17affyqUWJYQ== 2026-02-20T16:41:50.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOjphpUGZNAhAAN9XRfGqmQq17affyqUWJYQ=="}' 2026-02-20T16:41:50.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 59a76eec-2434-406f-bd98-2c41c3dcfe82 -i td/osd-scrub-repair/8/new.json 2026-02-20T16:41:50.199 INFO:tasks.workunit.client.0.trial013.stdout:8 2026-02-20T16:41:50.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-20T16:41:50.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOjphpUGZNAhAAN9XRfGqmQq17affyqUWJYQ== --osd-uuid 59a76eec-2434-406f-bd98-2c41c3dcfe82 2026-02-20T16:41:50.225 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:50.223+0000 7fee6c826a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:50.227 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:50.223+0000 7fee6c826a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:50.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:50.227+0000 7fee6c826a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:50.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:50.227+0000 7fee6c826a40 -1 bdev(0x55be6571ac00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:50.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:50.227+0000 7fee6c826a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-20T16:41:50.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-20T16:41:50.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:50.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-20T16:41:50.891 INFO:tasks.workunit.client.0.trial013.stdout:adding osd8 key to auth repository 2026-02-20T16:41:50.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:51.099 INFO:tasks.workunit.client.0.trial013.stdout:start osd.8 2026-02-20T16:41:51.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-20T16:41:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:51.116 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:51.115+0000 7f68704cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:51.117 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:51.115+0000 7f68704cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:51.118 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:51.115+0000 7f68704cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:51.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-20T16:41:51.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:51.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:41:51.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:51.597 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:51.595+0000 7f68704cba40 -1 Falling back to public interface 2026-02-20T16:41:52.253 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:52.251+0000 7f68704cba40 -1 osd.8 0 log_to_monitors true 2026-02-20T16:41:52.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:52.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:52.417 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:52.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:52.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:52.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:41:52.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:53.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-20T16:41:53.731 INFO:tasks.workunit.client.0.trial013.stdout:osd.8 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/711650503,v1:127.0.0.1:6867/711650503] [v2:127.0.0.1:6868/711650503,v1:127.0.0.1:6869/711650503] exists,up 59a76eec-2434-406f-bd98-2c41c3dcfe82 2026-02-20T16:41:53.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:53.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:53.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:53.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:41:53.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:41:53.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:41:53.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:41:53.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:41:53.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:41:53.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 b3f2296f-f3e9-4f70-91f3-adb3ac619a32' 2026-02-20T16:41:53.739 INFO:tasks.workunit.client.0.trial013.stdout:add osd9 b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:41:53.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:41:53.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRjphpMlesLBAAOjl7DAvP7990T8mcnozufg== 2026-02-20T16:41:53.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRjphpMlesLBAAOjl7DAvP7990T8mcnozufg=="}' 2026-02-20T16:41:53.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b3f2296f-f3e9-4f70-91f3-adb3ac619a32 -i td/osd-scrub-repair/9/new.json 2026-02-20T16:41:53.908 INFO:tasks.workunit.client.0.trial013.stdout:9 2026-02-20T16:41:53.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-20T16:41:53.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRjphpMlesLBAAOjl7DAvP7990T8mcnozufg== --osd-uuid b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:41:53.935 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:53.935+0000 7f497c645a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:53.937 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:53.935+0000 7f497c645a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:53.938 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:53.935+0000 7f497c645a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:53.939 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:53.935+0000 7f497c645a40 -1 bdev(0x55a1c4eaac00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-20T16:41:53.939 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:53.935+0000 7f497c645a40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-20T16:41:54.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-20T16:41:54.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:41:54.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-20T16:41:54.670 INFO:tasks.workunit.client.0.trial013.stdout:adding osd9 key to auth repository 2026-02-20T16:41:54.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:41:54.866 INFO:tasks.workunit.client.0.trial013.stdout:start osd.9 2026-02-20T16:41:54.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-20T16:41:54.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:41:54.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:41:54.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:41:54.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:41:54.891 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:54.887+0000 7ff05641da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:54.892 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:54.891+0000 7ff05641da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:54.893 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:54.891+0000 7ff05641da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:41:55.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:55.030 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:41:55.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:41:55.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:55.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:41:55.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:55.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:55.315+0000 7ff05641da40 -1 Falling back to public interface 2026-02-20T16:41:55.965 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:41:55.963+0000 7ff05641da40 -1 osd.9 0 log_to_monitors true 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:56.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:41:56.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:41:57.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:41:57.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:41:57.350 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:41:57.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:41:57.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:41:57.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stdout:osd.9 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/66965218,v1:127.0.0.1:6875/66965218] [v2:127.0.0.1:6876/66965218,v1:127.0.0.1:6877/66965218] exists,up b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-20T16:41:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:41:57.651 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:41:57.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:41:57.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:41:57.884 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:41:57.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:41:58.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:41:59.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-20T16:41:59.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:41:59.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:41:59.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:41:59.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:41:59.177 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:41:59.177 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:41:59.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:41:59.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:41:59.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:41:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:41:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:41:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:41:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:41:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:41:59.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:41:59.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-20T16:41:59.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-20T16:41:59.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-20T16:41:59.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:41:59.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-20T16:41:59.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-20T16:41:59.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-20T16:41:59.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:41:59.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-20T16:41:59.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-20T16:41:59.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2026-02-20T16:41:59.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:41:59.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378631 2026-02-20T16:41:59.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378631 2026-02-20T16:41:59.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631' 2026-02-20T16:41:59.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:41:59.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247814 2026-02-20T16:41:59.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247814 2026-02-20T16:41:59.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814' 2026-02-20T16:41:59.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:41:59.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-20T16:41:59.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-20T16:41:59.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814 5-115964116997' 2026-02-20T16:41:59.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:41:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-20T16:41:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-20T16:41:59.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814 5-115964116997 6-137438953476' 2026-02-20T16:41:59.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:41:59.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:00.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789956 2026-02-20T16:42:00.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789956 2026-02-20T16:42:00.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814 5-115964116997 6-137438953476 7-158913789956' 2026-02-20T16:42:00.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:00.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626435 2026-02-20T16:42:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626435 2026-02-20T16:42:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814 5-115964116997 6-137438953476 7-158913789956 8-180388626435' 2026-02-20T16:42:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:00.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:00.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495618 2026-02-20T16:42:00.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495618 2026-02-20T16:42:00.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-81604378631 4-98784247814 5-115964116997 6-137438953476 7-158913789956 8-180388626435 9-197568495618' 2026-02-20T16:42:00.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:00.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-20T16:42:00.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:00.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:00.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-20T16:42:00.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:00.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-20T16:42:00.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-20T16:42:00.191 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836489 2026-02-20T16:42:00.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:00.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-20T16:42:00.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:01.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:01.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:01.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-20T16:42:01.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:01.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-20T16:42:01.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:01.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:01.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-20T16:42:01.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:01.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-20T16:42:01.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-20T16:42:01.523 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672968 2026-02-20T16:42:01.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:01.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-20T16:42:01.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:01.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-20T16:42:01.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:01.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:01.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-20T16:42:01.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:01.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-20T16:42:01.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-20T16:42:01.683 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509447 2026-02-20T16:42:01.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:01.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-20T16:42:01.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:01.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378631 2026-02-20T16:42:01.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:01.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:01.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378631 2026-02-20T16:42:01.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:01.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378631 2026-02-20T16:42:01.852 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 81604378631 2026-02-20T16:42:01.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2026-02-20T16:42:01.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378631 -lt 81604378631 2026-02-20T16:42:02.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247814 2026-02-20T16:42:02.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:02.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247814 2026-02-20T16:42:02.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247814 2026-02-20T16:42:02.015 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.4 seq 98784247814 2026-02-20T16:42:02.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2026-02-20T16:42:02.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:02.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-20T16:42:02.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116997 2026-02-20T16:42:02.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:02.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116997 2026-02-20T16:42:02.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116997 2026-02-20T16:42:02.176 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.5 seq 115964116997 2026-02-20T16:42:02.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116997' 2026-02-20T16:42:02.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:02.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-20T16:42:02.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953476 2026-02-20T16:42:02.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:02.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953476 2026-02-20T16:42:02.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-20T16:42:02.345 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.6 seq 137438953476 2026-02-20T16:42:02.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953476' 2026-02-20T16:42:02.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:02.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953476 2026-02-20T16:42:02.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789956 2026-02-20T16:42:02.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:02.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789956 2026-02-20T16:42:02.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789956 2026-02-20T16:42:02.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789956' 2026-02-20T16:42:02.507 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.7 seq 158913789956 2026-02-20T16:42:02.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:02.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789956 -lt 158913789956 2026-02-20T16:42:02.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626435 2026-02-20T16:42:02.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:02.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626435 2026-02-20T16:42:02.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626435 2026-02-20T16:42:02.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626435' 2026-02-20T16:42:02.676 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.8 seq 180388626435 2026-02-20T16:42:02.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:02.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626435 -lt 180388626435 2026-02-20T16:42:02.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:02.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495618 2026-02-20T16:42:02.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:02.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:02.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495618 2026-02-20T16:42:02.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:02.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495618 2026-02-20T16:42:02.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495618' 2026-02-20T16:42:02.844 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.9 seq 197568495618 2026-02-20T16:42:02.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:03.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495619 -lt 197568495618 2026-02-20T16:42:03.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:03.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:03.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:03.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:42:03.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:03.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:03.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:03.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:03.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:03.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:03.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:03.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:03.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:03.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:03.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:03.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:42:03.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:03.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:42:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-20T16:42:03.764 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:42:03.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:42:03.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:42:04.069 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:42:04.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:42:05.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T16:42:05.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T16:42:05.275 INFO:tasks.workunit.client.0.trial013.stderr:set pool 2 allow_ec_overwrites to true 2026-02-20T16:42:05.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:42:05.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:05.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:05.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:05.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:05.287 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:05.287 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:42:05.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:05.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:05.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:05.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:05.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:05.559 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:05.559 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:05.559 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:42:05.559 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:05.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:42:05.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:42:05.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:42:05.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:05.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-20T16:42:05.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-20T16:42:05.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-20T16:42:05.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:05.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-20T16:42:05.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-20T16:42:05.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2026-02-20T16:42:05.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:05.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-20T16:42:05.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-20T16:42:05.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633' 2026-02-20T16:42:05.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:05.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247816 2026-02-20T16:42:05.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247816 2026-02-20T16:42:05.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816' 2026-02-20T16:42:05.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:05.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-20T16:42:05.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-20T16:42:05.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964116999' 2026-02-20T16:42:05.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:05.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:06.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953479 2026-02-20T16:42:06.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953479 2026-02-20T16:42:06.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964116999 6-137438953479' 2026-02-20T16:42:06.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:06.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:06.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789958 2026-02-20T16:42:06.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789958 2026-02-20T16:42:06.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964116999 6-137438953479 7-158913789958' 2026-02-20T16:42:06.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:06.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:06.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626437 2026-02-20T16:42:06.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626437 2026-02-20T16:42:06.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964116999 6-137438953479 7-158913789958 8-180388626437' 2026-02-20T16:42:06.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:06.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:06.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495620 2026-02-20T16:42:06.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495620 2026-02-20T16:42:06.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964116999 6-137438953479 7-158913789958 8-180388626437 9-197568495620' 2026-02-20T16:42:06.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:06.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:42:06.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:06.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:06.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:42:06.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:06.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:42:06.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:42:06.293 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T16:42:06.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:06.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:42:06.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:07.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:07.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:07.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:42:07.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:07.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-20T16:42:07.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:07.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:07.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-20T16:42:07.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:07.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-20T16:42:07.627 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672970 2026-02-20T16:42:07.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-20T16:42:07.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:07.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-20T16:42:07.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:07.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-20T16:42:07.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:07.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:07.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-20T16:42:07.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:07.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-20T16:42:07.791 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509449 2026-02-20T16:42:07.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-20T16:42:07.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:07.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509449 2026-02-20T16:42:07.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:07.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378633 2026-02-20T16:42:07.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:07.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:07.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378633 2026-02-20T16:42:07.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:07.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378633 2026-02-20T16:42:07.967 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 81604378633 2026-02-20T16:42:07.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2026-02-20T16:42:07.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:08.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-20T16:42:08.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247816 2026-02-20T16:42:08.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:08.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247816 2026-02-20T16:42:08.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247816 2026-02-20T16:42:08.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247816' 2026-02-20T16:42:08.140 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.4 seq 98784247816 2026-02-20T16:42:08.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:08.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247816 -lt 98784247816 2026-02-20T16:42:08.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116999 2026-02-20T16:42:08.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:08.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116999 2026-02-20T16:42:08.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-20T16:42:08.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116999' 2026-02-20T16:42:08.303 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.5 seq 115964116999 2026-02-20T16:42:08.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:08.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-20T16:42:08.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953479 2026-02-20T16:42:08.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:08.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953479 2026-02-20T16:42:08.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953479 2026-02-20T16:42:08.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953479' 2026-02-20T16:42:08.462 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.6 seq 137438953479 2026-02-20T16:42:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:08.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953479 -lt 137438953479 2026-02-20T16:42:08.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789958 2026-02-20T16:42:08.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:08.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789958 2026-02-20T16:42:08.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789958 2026-02-20T16:42:08.626 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.7 seq 158913789958 2026-02-20T16:42:08.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789958' 2026-02-20T16:42:08.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:08.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789958 -lt 158913789958 2026-02-20T16:42:08.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626437 2026-02-20T16:42:08.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:08.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626437 2026-02-20T16:42:08.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626437 2026-02-20T16:42:08.797 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.8 seq 180388626437 2026-02-20T16:42:08.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626437' 2026-02-20T16:42:08.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:08.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626438 -lt 180388626437 2026-02-20T16:42:08.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:08.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495620 2026-02-20T16:42:08.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:08.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:08.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495620 2026-02-20T16:42:08.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:08.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495620 2026-02-20T16:42:08.969 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.9 seq 197568495620 2026-02-20T16:42:08.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495620' 2026-02-20T16:42:08.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:09.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495621 -lt 197568495620 2026-02-20T16:42:09.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:09.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:09.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:09.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:09.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:09.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:09.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:09.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:09.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:42:09.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:42:09.885 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:42:09.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:42:10.090 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:42:10.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:42:10.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:42:10.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:42:10.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T16:42:10.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T16:42:10.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:42:10.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:42:10.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:42:10.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-20T16:42:10.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T16:42:10.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:42:10.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T16:42:10.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-20T16:42:10.292 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:42:10.293 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:9 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr:7' 2026-02-20T16:42:10.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:42:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:10.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:10.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:42:10.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:42:10.971 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:42:11.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:42:11.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:11.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:11.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:11.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:11.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:42:11.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:42:11.102 INFO:tasks.workunit.client.0.trial013.stderr:start osd.3 2026-02-20T16:42:11.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:11.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:42:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:42:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:11.116 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:11.114+0000 7f24011b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:11.118 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:11.114+0000 7f24011b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:11.119 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:11.114+0000 7f24011b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:11.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:42:11.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:11.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:42:11.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:11.541 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:11.538+0000 7f24011b4a40 -1 Falling back to public interface 2026-02-20T16:42:12.150 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:12.146+0000 7f24011b4a40 -1 osd.3 64 log_to_monitors true 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:12.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:42:12.584 INFO:tasks.workunit.client.0.trial013.stderr:osd.3 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [19,64) [v2:127.0.0.1:6826/3315263495,v1:127.0.0.1:6827/3315263495] [v2:127.0.0.1:6828/3315263495,v1:127.0.0.1:6829/3315263495] exists,up f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:42:12.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:12.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:12.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:12.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:12.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:12.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:12.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:12.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:12.586 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:12.586 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:42:12.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:12.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:12.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:12.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:12.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:12.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:12.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:12.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:12.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:12.867 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:42:12.868 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:42:12.868 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:42:12.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:12.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:12.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:12.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:42:12.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:42:12.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:42:12.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:12.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-20T16:42:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-20T16:42:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2026-02-20T16:42:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:13.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509452 2026-02-20T16:42:13.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509452 2026-02-20T16:42:13.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452' 2026-02-20T16:42:13.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776130 2026-02-20T16:42:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776130 2026-02-20T16:42:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130' 2026-02-20T16:42:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:13.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247818 2026-02-20T16:42:13.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247818 2026-02-20T16:42:13.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818' 2026-02-20T16:42:13.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117002 2026-02-20T16:42:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117002 2026-02-20T16:42:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818 5-115964117002' 2026-02-20T16:42:13.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:13.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953481 2026-02-20T16:42:13.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953481 2026-02-20T16:42:13.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818 5-115964117002 6-137438953481' 2026-02-20T16:42:13.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789960 2026-02-20T16:42:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789960 2026-02-20T16:42:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818 5-115964117002 6-137438953481 7-158913789960' 2026-02-20T16:42:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:13.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626440 2026-02-20T16:42:13.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626440 2026-02-20T16:42:13.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818 5-115964117002 6-137438953481 7-158913789960 8-180388626440' 2026-02-20T16:42:13.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:13.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:13.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495623 2026-02-20T16:42:13.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495623 2026-02-20T16:42:13.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-292057776130 4-98784247818 5-115964117002 6-137438953481 7-158913789960 8-180388626440 9-197568495623' 2026-02-20T16:42:13.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:13.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:42:13.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:13.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:13.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:42:13.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:13.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:42:13.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:42:13.603 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T16:42:13.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:13.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-20T16:42:13.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:14.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:14.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:14.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-20T16:42:14.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:14.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-20T16:42:14.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:14.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:14.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672973 2026-02-20T16:42:14.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:14.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672973 2026-02-20T16:42:14.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-20T16:42:14.932 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672973 2026-02-20T16:42:14.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:15.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672973 -lt 42949672973 2026-02-20T16:42:15.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509452 2026-02-20T16:42:15.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:15.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509452 2026-02-20T16:42:15.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509452 2026-02-20T16:42:15.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2026-02-20T16:42:15.095 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509452 2026-02-20T16:42:15.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:15.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509452 2026-02-20T16:42:15.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776130 2026-02-20T16:42:15.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:15.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776130 2026-02-20T16:42:15.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776130 2026-02-20T16:42:15.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776130' 2026-02-20T16:42:15.260 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 292057776130 2026-02-20T16:42:15.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:15.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776130 -lt 292057776130 2026-02-20T16:42:15.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247818 2026-02-20T16:42:15.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247818 2026-02-20T16:42:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247818 2026-02-20T16:42:15.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247818' 2026-02-20T16:42:15.430 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247818 2026-02-20T16:42:15.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:15.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247819 -lt 98784247818 2026-02-20T16:42:15.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117002 2026-02-20T16:42:15.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:15.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117002 2026-02-20T16:42:15.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117002 2026-02-20T16:42:15.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117002' 2026-02-20T16:42:15.600 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 115964117002 2026-02-20T16:42:15.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:15.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117002 -lt 115964117002 2026-02-20T16:42:15.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953481 2026-02-20T16:42:15.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:15.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953481 2026-02-20T16:42:15.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953481 2026-02-20T16:42:15.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953481' 2026-02-20T16:42:15.769 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953481 2026-02-20T16:42:15.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:15.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953481 -lt 137438953481 2026-02-20T16:42:15.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:15.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789960 2026-02-20T16:42:15.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:15.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:15.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789960 2026-02-20T16:42:15.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:15.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789960 2026-02-20T16:42:15.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789960' 2026-02-20T16:42:15.933 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 158913789960 2026-02-20T16:42:15.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:16.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789961 -lt 158913789960 2026-02-20T16:42:16.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626440 2026-02-20T16:42:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:16.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:16.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626440 2026-02-20T16:42:16.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:16.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626440 2026-02-20T16:42:16.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626440' 2026-02-20T16:42:16.109 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 180388626440 2026-02-20T16:42:16.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:16.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626440 -lt 180388626440 2026-02-20T16:42:16.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:16.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495623 2026-02-20T16:42:16.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:16.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:16.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495623 2026-02-20T16:42:16.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:16.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495623 2026-02-20T16:42:16.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495623' 2026-02-20T16:42:16.278 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 197568495623 2026-02-20T16:42:16.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:16.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495623 -lt 197568495623 2026-02-20T16:42:16.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:16.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:16.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:16.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:16.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:16.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:16.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:16.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:16.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:16.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:16.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:16.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:16.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:16.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:16.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:42:16.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:42:16.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:42:16.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:42:17.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:42:17.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:42:17.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:42:17.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:42:17.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:17.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:17.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:17.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:17.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:17.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:42:17.451 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:17.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:17.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:17.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:17.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:17.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:17.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:04.068627+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:17.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:18.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:18.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:18.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:18.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:18.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:18.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:18.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:18.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:04.068627+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:18.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:19.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:19.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:19.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:19.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:19.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:19.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:19.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:19.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:04.068627+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:19.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:20.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:20.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:20.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:20.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:20.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:20.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:20.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:21.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:04.068627+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:21.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:22.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:22.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:22.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:22.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:22.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:22.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:22.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:22.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:04.068627+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:22.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:23.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:23.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:23.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:23.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:23.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:23.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:23.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:23.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:18.151375+0000 '>' 2026-02-20T16:42:04.068627+0000 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:42:23.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:23.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:23.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:23.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:23.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:42:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:42:23.814 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:42:23.815 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:42:23.815 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:23.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:23.918 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:23.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:23.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:start osd.9 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:23.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:23.936 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:23.934+0000 7f2f8ff91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:23.937 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:23.934+0000 7f2f8ff91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:23.939 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:23.934+0000 7f2f8ff91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:24.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:42:24.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:24.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:24.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:24.369 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:24.366+0000 7f2f8ff91a40 -1 Falling back to public interface 2026-02-20T16:42:25.117 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:25.114+0000 7f2f8ff91a40 -1 osd.9 71 log_to_monitors true 2026-02-20T16:42:25.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:25.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:25.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:25.239 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:25.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:25.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:25.403 INFO:tasks.workunit.client.0.trial013.stderr:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 71 last_clean_interval [46,71) [v2:127.0.0.1:6874/1187571676,v1:127.0.0.1:6875/1187571676] [v2:127.0.0.1:6876/1187571676,v1:127.0.0.1:6877/1187571676] exists,up b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:42:25.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:25.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:25.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:25.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:25.405 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:25.405 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:42:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:25.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:25.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:25.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:25.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:25.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:25.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:25.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:25.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:25.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:25.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-20T16:42:25.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-20T16:42:25.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-20T16:42:25.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:25.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-20T16:42:25.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-20T16:42:25.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976' 2026-02-20T16:42:25.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:25.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:25.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-20T16:42:25.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-20T16:42:25.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455' 2026-02-20T16:42:25.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:25.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:25.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776133 2026-02-20T16:42:25.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776133 2026-02-20T16:42:25.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133' 2026-02-20T16:42:25.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:25.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:26.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247822 2026-02-20T16:42:26.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247822 2026-02-20T16:42:26.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822' 2026-02-20T16:42:26.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:26.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:26.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-20T16:42:26.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-20T16:42:26.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822 5-115964117005' 2026-02-20T16:42:26.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:26.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:26.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953485 2026-02-20T16:42:26.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953485 2026-02-20T16:42:26.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822 5-115964117005 6-137438953485' 2026-02-20T16:42:26.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:26.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:26.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789964 2026-02-20T16:42:26.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789964 2026-02-20T16:42:26.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822 5-115964117005 6-137438953485 7-158913789964' 2026-02-20T16:42:26.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:26.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:26.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626443 2026-02-20T16:42:26.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626443 2026-02-20T16:42:26.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822 5-115964117005 6-137438953485 7-158913789964 8-180388626443' 2026-02-20T16:42:26.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:26.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:26.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579906 2026-02-20T16:42:26.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579906 2026-02-20T16:42:26.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-292057776133 4-98784247822 5-115964117005 6-137438953485 7-158913789964 8-180388626443 9-317827579906' 2026-02-20T16:42:26.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:26.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-20T16:42:26.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:26.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:26.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-20T16:42:26.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:26.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-20T16:42:26.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-20T16:42:26.399 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836497 2026-02-20T16:42:26.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:26.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-20T16:42:26.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:27.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:27.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-20T16:42:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:27.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-20T16:42:27.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:27.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:27.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-20T16:42:27.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:27.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-20T16:42:27.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-20T16:42:27.734 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672976 2026-02-20T16:42:27.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:27.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-20T16:42:27.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:27.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-20T16:42:27.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:27.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:27.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-20T16:42:27.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:27.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-20T16:42:27.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-20T16:42:27.904 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509455 2026-02-20T16:42:27.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:28.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509456 -lt 64424509455 2026-02-20T16:42:28.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776133 2026-02-20T16:42:28.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:28.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776133 2026-02-20T16:42:28.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776133 2026-02-20T16:42:28.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776133' 2026-02-20T16:42:28.067 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 292057776133 2026-02-20T16:42:28.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:28.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776133 -lt 292057776133 2026-02-20T16:42:28.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247822 2026-02-20T16:42:28.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:28.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247822 2026-02-20T16:42:28.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247822 2026-02-20T16:42:28.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247822' 2026-02-20T16:42:28.243 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247822 2026-02-20T16:42:28.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:28.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247822 -lt 98784247822 2026-02-20T16:42:28.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117005 2026-02-20T16:42:28.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:28.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117005 2026-02-20T16:42:28.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117005 2026-02-20T16:42:28.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117005' 2026-02-20T16:42:28.406 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 115964117005 2026-02-20T16:42:28.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:28.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-20T16:42:28.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953485 2026-02-20T16:42:28.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:28.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953485 2026-02-20T16:42:28.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953485 2026-02-20T16:42:28.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953485' 2026-02-20T16:42:28.584 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953485 2026-02-20T16:42:28.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:28.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953485 -lt 137438953485 2026-02-20T16:42:28.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789964 2026-02-20T16:42:28.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:28.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789964 2026-02-20T16:42:28.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789964 2026-02-20T16:42:28.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789964' 2026-02-20T16:42:28.746 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 158913789964 2026-02-20T16:42:28.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:28.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789964 -lt 158913789964 2026-02-20T16:42:28.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:28.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626443 2026-02-20T16:42:28.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:28.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:28.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626443 2026-02-20T16:42:28.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:28.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626443 2026-02-20T16:42:28.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626443' 2026-02-20T16:42:28.905 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 180388626443 2026-02-20T16:42:28.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:29.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626444 -lt 180388626443 2026-02-20T16:42:29.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:29.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-317827579906 2026-02-20T16:42:29.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:29.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:29.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-317827579906 2026-02-20T16:42:29.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:29.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=317827579906 2026-02-20T16:42:29.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 317827579906' 2026-02-20T16:42:29.083 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 317827579906 2026-02-20T16:42:29.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:29.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 317827579906 -lt 317827579906 2026-02-20T16:42:29.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:29.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:29.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:29.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:29.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:29.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:29.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:29.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:29.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:29.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:42:29.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:42:29.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:42:29.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:29.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:29.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:29.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:29.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:29.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:29.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:30.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:30.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:30.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:30.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:30.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:42:30.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:42:30.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:42:30.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:42:30.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:42:30.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:42:30.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:30.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:30.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:30.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:30.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:30.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:30.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:30.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:30.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:42:31.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:42:31.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-20T16:42:31.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:42:31.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-20T16:42:31.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:42:31.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:31.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:42:31.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:42:31.510 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:42:31.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:42:31.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:31.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:42:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:31.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:start osd.5 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:31.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:31.646 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:31.642+0000 7f57b688fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:31.648 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:31.646+0000 7f57b688fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:31.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:31.646+0000 7f57b688fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:31.790 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:42:31.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:31.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:42:31.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:32.097 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:32.094+0000 7f57b688fa40 -1 Falling back to public interface 2026-02-20T16:42:32.742 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:32.738+0000 7f57b688fa40 -1 osd.5 77 log_to_monitors true 2026-02-20T16:42:32.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:32.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:32.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:32.952 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:32.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:32.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:osd.5 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [27,77) [v2:127.0.0.1:6842/1414662416,v1:127.0.0.1:6843/1414662416] [v2:127.0.0.1:6844/1414662416,v1:127.0.0.1:6845/1414662416] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:33.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:33.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:33.110 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:33.111 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:42:33.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:33.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:33.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:33.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:33.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:33.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:33.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:33.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:33.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:33.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:33.385 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:33.385 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:33.385 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:33.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-20T16:42:33.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-20T16:42:33.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-20T16:42:33.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672979 2026-02-20T16:42:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672979 2026-02-20T16:42:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979' 2026-02-20T16:42:33.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:33.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509458 2026-02-20T16:42:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509458 2026-02-20T16:42:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458' 2026-02-20T16:42:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:33.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776136 2026-02-20T16:42:33.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776136 2026-02-20T16:42:33.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136' 2026-02-20T16:42:33.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:33.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247825 2026-02-20T16:42:33.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247825 2026-02-20T16:42:33.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825' 2026-02-20T16:42:33.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:33.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-20T16:42:33.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-20T16:42:33.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825 5-343597383682' 2026-02-20T16:42:33.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:33.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953487 2026-02-20T16:42:33.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953487 2026-02-20T16:42:33.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825 5-343597383682 6-137438953487' 2026-02-20T16:42:33.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789966 2026-02-20T16:42:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789966 2026-02-20T16:42:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825 5-343597383682 6-137438953487 7-158913789966' 2026-02-20T16:42:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:33.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:34.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626446 2026-02-20T16:42:34.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626446 2026-02-20T16:42:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825 5-343597383682 6-137438953487 7-158913789966 8-180388626446' 2026-02-20T16:42:34.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:34.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:34.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579908 2026-02-20T16:42:34.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579908 2026-02-20T16:42:34.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509458 3-292057776136 4-98784247825 5-343597383682 6-137438953487 7-158913789966 8-180388626446 9-317827579908' 2026-02-20T16:42:34.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:34.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-20T16:42:34.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:34.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:34.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-20T16:42:34.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:34.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-20T16:42:34.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-20T16:42:34.111 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836499 2026-02-20T16:42:34.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:34.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-20T16:42:34.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:35.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:35.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:35.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-20T16:42:35.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:35.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672979 2026-02-20T16:42:35.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:35.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:35.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672979 2026-02-20T16:42:35.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:35.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672979 2026-02-20T16:42:35.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2026-02-20T16:42:35.438 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672979 2026-02-20T16:42:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:35.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672979 -lt 42949672979 2026-02-20T16:42:35.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:35.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509458 2026-02-20T16:42:35.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:35.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:35.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509458 2026-02-20T16:42:35.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:35.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509458 2026-02-20T16:42:35.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2026-02-20T16:42:35.606 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509458 2026-02-20T16:42:35.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:35.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509458 -lt 64424509458 2026-02-20T16:42:35.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:35.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776136 2026-02-20T16:42:35.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:35.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:35.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776136 2026-02-20T16:42:35.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:35.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776136 2026-02-20T16:42:35.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776136' 2026-02-20T16:42:35.767 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 292057776136 2026-02-20T16:42:35.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:35.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776136 -lt 292057776136 2026-02-20T16:42:35.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:35.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247825 2026-02-20T16:42:35.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:35.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:35.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247825 2026-02-20T16:42:35.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:35.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247825 2026-02-20T16:42:35.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247825' 2026-02-20T16:42:35.938 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247825 2026-02-20T16:42:35.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:36.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247825 -lt 98784247825 2026-02-20T16:42:36.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:36.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-343597383682 2026-02-20T16:42:36.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:36.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:36.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-343597383682 2026-02-20T16:42:36.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:36.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-20T16:42:36.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 343597383682' 2026-02-20T16:42:36.106 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 343597383682 2026-02-20T16:42:36.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:36.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383682 -lt 343597383682 2026-02-20T16:42:36.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953487 2026-02-20T16:42:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:36.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:36.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953487 2026-02-20T16:42:36.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:36.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953487 2026-02-20T16:42:36.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953487' 2026-02-20T16:42:36.287 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953487 2026-02-20T16:42:36.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:36.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953487 -lt 137438953487 2026-02-20T16:42:36.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:36.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789966 2026-02-20T16:42:36.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:36.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:36.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789966 2026-02-20T16:42:36.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:36.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789966 2026-02-20T16:42:36.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789966' 2026-02-20T16:42:36.450 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 158913789966 2026-02-20T16:42:36.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:36.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789967 -lt 158913789966 2026-02-20T16:42:36.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:36.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626446 2026-02-20T16:42:36.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:36.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:36.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626446 2026-02-20T16:42:36.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:36.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626446 2026-02-20T16:42:36.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626446' 2026-02-20T16:42:36.630 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 180388626446 2026-02-20T16:42:36.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:36.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626446 -lt 180388626446 2026-02-20T16:42:36.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:36.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-317827579908 2026-02-20T16:42:36.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:36.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:36.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-317827579908 2026-02-20T16:42:36.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:36.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=317827579908 2026-02-20T16:42:36.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 317827579908' 2026-02-20T16:42:36.790 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 317827579908 2026-02-20T16:42:36.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:36.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 317827579909 -lt 317827579908 2026-02-20T16:42:36.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:36.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:36.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:37.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:37.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:37.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:37.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:37.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:37.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:37.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:37.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:37.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:37.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:37.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-20T16:42:37.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:42:37.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:42:37.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:42:37.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:42:37.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-20T16:42:37.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-20T16:42:37.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:42:37.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:42:37.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:37.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:37.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:37.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:37.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:37.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:42:37.957 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:42:37.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:37.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:37.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:37.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:37.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:37.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:38.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:18.151375+0000 '>' 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:38.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:39.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:39.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:39.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:39.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:39.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:39.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:39.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:39.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:18.151375+0000 '>' 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:39.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:40.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:40.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:40.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:40.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:40.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:40.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:18.151375+0000 '>' 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:40.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:41.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:41.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:41.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:41.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:18.151375+0000 '>' 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:41.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:42.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:42.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:42.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:42.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:42.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:42.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:42.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:42.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:38.913151+0000 '>' 2026-02-20T16:42:18.151375+0000 2026-02-20T16:42:42.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:42:42.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:42.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:42.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:42.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:43.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:42:43.157 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:42:43.157 INFO:tasks.workunit.client.0.trial013.stdout:hinfo_key 2026-02-20T16:42:43.157 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:42:43.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:43.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:43.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:43.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:start osd.9 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:43.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:43.280 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:43.278+0000 7f76e88f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:43.282 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:43.278+0000 7f76e88f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:43.282 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:43.278+0000 7f76e88f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:43.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:43.716 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:43.714+0000 7f76e88f1a40 -1 Falling back to public interface 2026-02-20T16:42:44.306 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:42:44.302+0000 7f76e88f1a40 -1 osd.9 81 log_to_monitors true 2026-02-20T16:42:44.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:44.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:44.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:44.581 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:44.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:44.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:44.733 INFO:tasks.workunit.client.0.trial013.stderr:osd.9 up in weight 1 up_from 85 up_thru 0 down_at 82 last_clean_interval [74,81) [v2:127.0.0.1:6874/1886040865,v1:127.0.0.1:6875/1886040865] [v2:127.0.0.1:6876/1886040865,v1:127.0.0.1:6877/1886040865] exists,up b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:44.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:44.735 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:44.735 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:42:44.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:44.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:44.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:44.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:45.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:45.009 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:42:45.009 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T16:42:45.009 INFO:tasks.workunit.client.0.trial013.stderr:3 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:4 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:5 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:6 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:7 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:8 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:9' 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:45.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-20T16:42:45.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-20T16:42:45.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-20T16:42:45.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:45.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672982 2026-02-20T16:42:45.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672982 2026-02-20T16:42:45.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982' 2026-02-20T16:42:45.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:45.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509461 2026-02-20T16:42:45.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509461 2026-02-20T16:42:45.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461' 2026-02-20T16:42:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:45.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776139 2026-02-20T16:42:45.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776139 2026-02-20T16:42:45.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139' 2026-02-20T16:42:45.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:45.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247828 2026-02-20T16:42:45.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247828 2026-02-20T16:42:45.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828' 2026-02-20T16:42:45.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383685 2026-02-20T16:42:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383685 2026-02-20T16:42:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828 5-343597383685' 2026-02-20T16:42:45.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:45.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953490 2026-02-20T16:42:45.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953490 2026-02-20T16:42:45.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828 5-343597383685 6-137438953490' 2026-02-20T16:42:45.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:45.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789970 2026-02-20T16:42:45.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789970 2026-02-20T16:42:45.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828 5-343597383685 6-137438953490 7-158913789970' 2026-02-20T16:42:45.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:45.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626449 2026-02-20T16:42:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626449 2026-02-20T16:42:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828 5-343597383685 6-137438953490 7-158913789970 8-180388626449' 2026-02-20T16:42:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:45.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220162 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220162 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-64424509461 3-292057776139 4-98784247828 5-343597383685 6-137438953490 7-158913789970 8-180388626449 9-365072220162' 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-20T16:42:45.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:45.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:45.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-20T16:42:45.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:45.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-20T16:42:45.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-20T16:42:45.738 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836503 2026-02-20T16:42:45.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:45.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-20T16:42:45.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:46.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:46.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:47.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836503 2026-02-20T16:42:47.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672982 2026-02-20T16:42:47.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:47.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672982 2026-02-20T16:42:47.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672982 2026-02-20T16:42:47.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2026-02-20T16:42:47.069 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 42949672982 2026-02-20T16:42:47.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:47.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672982 -lt 42949672982 2026-02-20T16:42:47.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509461 2026-02-20T16:42:47.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:47.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509461 2026-02-20T16:42:47.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509461 2026-02-20T16:42:47.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2026-02-20T16:42:47.238 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509461 2026-02-20T16:42:47.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:47.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509462 -lt 64424509461 2026-02-20T16:42:47.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776139 2026-02-20T16:42:47.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:47.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776139 2026-02-20T16:42:47.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776139 2026-02-20T16:42:47.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776139' 2026-02-20T16:42:47.400 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.3 seq 292057776139 2026-02-20T16:42:47.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:47.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776139 -lt 292057776139 2026-02-20T16:42:47.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247828 2026-02-20T16:42:47.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:47.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247828 2026-02-20T16:42:47.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247828 2026-02-20T16:42:47.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247828' 2026-02-20T16:42:47.562 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.4 seq 98784247828 2026-02-20T16:42:47.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:47.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247828 -lt 98784247828 2026-02-20T16:42:47.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-343597383685 2026-02-20T16:42:47.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:47.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-343597383685 2026-02-20T16:42:47.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383685 2026-02-20T16:42:47.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 343597383685' 2026-02-20T16:42:47.734 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.5 seq 343597383685 2026-02-20T16:42:47.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:47.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383685 -lt 343597383685 2026-02-20T16:42:47.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:47.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953490 2026-02-20T16:42:47.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:47.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:47.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953490 2026-02-20T16:42:47.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:47.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953490 2026-02-20T16:42:47.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953490' 2026-02-20T16:42:47.895 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.6 seq 137438953490 2026-02-20T16:42:47.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:48.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953491 -lt 137438953490 2026-02-20T16:42:48.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:48.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789970 2026-02-20T16:42:48.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:48.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789970 2026-02-20T16:42:48.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:48.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789970 2026-02-20T16:42:48.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789970' 2026-02-20T16:42:48.066 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.7 seq 158913789970 2026-02-20T16:42:48.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:48.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789970 -lt 158913789970 2026-02-20T16:42:48.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:48.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626449 2026-02-20T16:42:48.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:48.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:48.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626449 2026-02-20T16:42:48.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:48.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626449 2026-02-20T16:42:48.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626449' 2026-02-20T16:42:48.238 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.8 seq 180388626449 2026-02-20T16:42:48.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:48.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626449 -lt 180388626449 2026-02-20T16:42:48.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:48.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-365072220162 2026-02-20T16:42:48.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:48.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:48.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-365072220162 2026-02-20T16:42:48.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:48.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=365072220162 2026-02-20T16:42:48.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 365072220162' 2026-02-20T16:42:48.418 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.9 seq 365072220162 2026-02-20T16:42:48.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:48.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220162 -lt 365072220162 2026-02-20T16:42:48.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:48.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:48.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:48.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:48.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:48.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:48.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:48.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:48.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:48.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:48.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:48.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:48.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:48.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:48.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:49.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:42:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-20T16:42:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:42:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-20T16:42:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:42:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:42:49.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:42:49.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:49.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:49.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:49.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:49.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:49.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:49.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:49.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:49.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:49.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:49.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:42:49.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:42:49.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:42:49.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:42:49.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:42:49.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:42:49.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:42:49.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:42:49.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:42:49.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:42:49.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:42:49.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:49.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:49.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:49.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:49.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:49.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:49.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:50.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:42:50.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:50.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:50.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:42:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:42:50.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:42:50.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:42:50.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:42:50.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:42:50.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:42:50.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:42:50.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:42:50.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:42:50.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:42:50.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:50.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:50.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:50.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:50.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:50.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:50.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:51.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:51.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:51.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:51.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:51.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:51.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:51.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:51.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:42:51.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-20T16:42:51.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224151"' 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224151' 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224152"' 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224152' 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 224151 224152' 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224151 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:51.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224154: /' 2026-02-20T16:42:51.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:42:51.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224156: /' 2026-02-20T16:42:51.951 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:51.952 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-20T16:42:51.953 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:51.954 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:51.955 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:51.955 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:51.955 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:42:51.955 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:42:51.955 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:51.956 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: start osd.9 2026-02-20T16:42:51.957 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:2224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:42:52.000 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:42:52.001 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:42:52.002 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:42:52.003 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:42:52.003 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:42:52.003 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:52.004 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:42:52.005 INFO:tasks.workunit.client.0.trial013.stderr:224154: start osd.5 2026-02-20T16:42:52.006 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:42:53.441 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2026-02-20T16:42:51.970+0000 7f94fa59ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2026-02-20T16:42:51.970+0000 7f94fa59ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2026-02-20T16:42:51.974+0000 7f94fa59ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:53.442 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: 0 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2026-02-20T16:42:52.470+0000 7f94fa59ba40 -1 Falling back to public interface 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2026-02-20T16:42:53.161+0000 7f94fa59ba40 -1 osd.9 86 log_to_monitors true 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:53.443 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: 1 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [85,86) [v2:127.0.0.1:6842/3863711429,v1:127.0.0.1:6843/3863711429] [v2:127.0.0.1:6844/3863711429,v1:127.0.0.1:6845/3863711429] exists,up b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:53.444 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:53.445 INFO:tasks.workunit.client.0.trial013.stderr:224156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:53.493 INFO:tasks.workunit.client.0.trial013.stderr:224156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge24154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:42:53.493 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2026-02-20T16:42:52.014+0000 7f753b78ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2026-02-20T16:42:52.014+0000 7f753b78ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2026-02-20T16:42:52.018+0000 7f753b78ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:42:53.494 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: 0 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2026-02-20T16:42:52.506+0000 7f753b78ba40 -1 Falling back to public interface 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2026-02-20T16:42:53.225+0000 7f753b78ba40 -1 osd.5 86 log_to_monitors true 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: 1 2026-02-20T16:42:53.495 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: osd.5 up in weight 1 up_from 91 up_thru 80 down_at 87 last_clean_interval [80,86) [v2:127.0.0.1:6874/3008869805,v1:127.0.0.1:6875/3008869805] [v2:127.0.0.1:6876/3008869805,v1:127.0.0.1:6877/3008869805] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:42:53.496 INFO:tasks.workunit.client.0.trial013.stderr:224154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:42:54.003 INFO:tasks.workunit.client.0.trial013.stderr:224154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gt_timeout_delays: echo true 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:54.004 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 1 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 2 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 3 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 4 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 5 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 6 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 7 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 8 2026-02-20T16:42:54.005 INFO:tasks.workunit.client.0.trial013.stderr:224156: 9' 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672985 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672985 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985' 2026-02-20T16:42:54.006 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509464 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509464 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464' 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776142 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776142 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142' 2026-02-20T16:42:54.007 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.055 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: et_timeout_delays: echo true 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:42:54.056 INFO:tasks.workunit.client.0.trial013.stderr:224154: 1 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 2 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 3 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 4 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 5 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 6 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 7 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 8 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: 9' 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-20T16:42:54.057 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506' 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672986 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672986 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986' 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509465 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509465 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465' 2026-02-20T16:42:54.058 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.059 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:42:54.059 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776143 2026-02-20T16:42:54.059 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776143 2026-02-20T16:42:54.059 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143' 2026-02-20T16:42:54.059 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.445 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247831 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247831 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831' 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023938 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023938 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831 5-390842023938' 2026-02-20T16:42:54.446 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953493 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953493 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831 5-390842023938 6-137438953493' 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789972 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789972 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831 5-390842023938 6-137438953493 7-158913789972' 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:54.447 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626452 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626452 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831 5-390842023938 6-137438953493 7-158913789972 8-180388626452' 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-292057776142 4-98784247831 5-390842023938 6-137438953493 7-158913789972 8-180388626452 9-386547056642' 2026-02-20T16:42:54.448 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/ceph flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247832 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247832 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832' 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023939 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023939 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832 5-390842023939' 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953494 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953494 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832 5-390842023939 6-137438953494' 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789973 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789973 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832 5-390842023939 6-137438953494 7-158913789973' 2026-02-20T16:42:54.488 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626453 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626453 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832 5-390842023939 6-137438953494 7-158913789973 8-180388626453' 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056643 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056643 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672986 2-64424509465 3-292057776143 4-98784247832 5-390842023939 6-137438953494 7-158913789973 8-180388626453 9-386547056643' 2026-02-20T16:42:54.489 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/ceptest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.0 seq 21474836505 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:56.097 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836505 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836505 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672985 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672985 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.098 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672985 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.1 seq 42949672985 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672985 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509464 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509464 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509464 2026-02-20T16:42:56.099 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2026-02-20T16:42:56.100 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.2 seq 64424509464 2026-02-20T16:42:56.100 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:56.100 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509465 -lt 64424509464 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_statshtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836506 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836506 2026-02-20T16:42:56.151 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.0 seq 21474836506 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836506 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836506 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672986 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:42:56.152 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672986 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672986 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.1 seq 42949672986 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672986 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509465 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509465 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.153 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509465 2026-02-20T16:42:56.154 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2026-02-20T16:42:56.154 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.2 seq 64424509465 2026-02-20T16:42:56.154 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:42:56.154 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509465 -lt 64424509465 2026-02-20T16:42:56.605 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stat: for s in $seqs 2026-02-20T16:42:56.605 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776142 2026-02-20T16:42:56.605 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776142 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776142 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776142' 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.3 seq 292057776142 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776143 -lt 292057776142 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247831 2026-02-20T16:42:56.606 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247831 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247831 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247831' 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.4 seq 98784247831 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247832 -lt 98784247831 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-390842023938 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.607 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-390842023938 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=390842023938 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 390842023938' 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.5 seq 390842023938 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 390842023939 -lt 390842023938 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953493 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.608 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:56.649 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-s: for s in $seqs 2026-02-20T16:42:56.649 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776143 2026-02-20T16:42:56.649 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.649 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776143 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776143 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776143' 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.3 seq 292057776143 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776143 -lt 292057776143 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247832 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:42:56.650 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247832 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247832 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247832' 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.4 seq 98784247832 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247832 -lt 98784247832 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-390842023939 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-390842023939 2026-02-20T16:42:56.651 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=390842023939 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 390842023939' 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.5 seq 390842023939 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 390842023939 -lt 390842023939 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953494 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:56.652 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:42:57.116 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cephhelpers.sh:2268: flush_pg_stats: echo 6-137438953493 2026-02-20T16:42:57.116 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.116 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953493 2026-02-20T16:42:57.116 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953493' 2026-02-20T16:42:57.116 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.6 seq 137438953493 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953495 -lt 137438953493 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789972 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789972 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789972 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789972' 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.7 seq 158913789972 2026-02-20T16:42:57.117 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789974 -lt 158913789972 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626452 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626452 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626452 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626452' 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.8 seq 180388626452 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626453 -lt 180388626452 2026-02-20T16:42:57.118 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056642 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056642 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.119 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-20T16:42:57.135 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 38654705-helpers.sh:2268: flush_pg_stats: echo 6-137438953494 2026-02-20T16:42:57.135 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.135 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953494 2026-02-20T16:42:57.135 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953494' 2026-02-20T16:42:57.135 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.6 seq 137438953494 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953495 -lt 137438953494 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789973 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789973 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789973 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789973' 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.7 seq 158913789973 2026-02-20T16:42:57.136 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789974 -lt 158913789973 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626453 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626453 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626453 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626453' 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.8 seq 180388626453 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:42:57.137 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626453 -lt 180388626453 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056643 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056643 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:42:57.138 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056643 2026-02-20T16:42:57.838 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 38654706642' 2026-02-20T16:42:57.838 INFO:tasks.workunit.client.0.trial013.stderr:224156: waiting osd.9 seq 386547056642 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056642 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:57.839 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:224156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:57.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:42:57.850 INFO:tasks.workunit.client.0.trial013.stderr:56643' 2026-02-20T16:42:57.850 INFO:tasks.workunit.client.0.trial013.stderr:224154: waiting osd.9 seq 386547056643 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056643 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:42:57.851 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:42:57.852 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:42:57.853 INFO:tasks.workunit.client.0.trial013.stderr:224154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:42:57.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:42:57.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:42:57.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224152 2026-02-20T16:42:57.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:42:57.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:42:58.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:42:58.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:42:58.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:42:58.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:42:58.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:58.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:58.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:58.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:58.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:42:38.913151+0000 2026-02-20T16:42:58.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:42:58.325 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:42:58.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:42:38.913151+0000 2026-02-20T16:42:58.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:42:58.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:42:38.913151+0000 2026-02-20T16:42:58.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:42:58.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:58.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:58.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:38.913151+0000 '>' 2026-02-20T16:42:38.913151+0000 2026-02-20T16:42:58.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:42:59.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:42:59.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:42:59.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:42:59.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:42:59.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:42:59.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:42:59.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:42:59.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:38.913151+0000 '>' 2026-02-20T16:42:38.913151+0000 2026-02-20T16:42:59.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:00.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:00.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:00.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:00.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:00.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:00.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:00.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:38.913151+0000 '>' 2026-02-20T16:42:38.913151+0000 2026-02-20T16:43:00.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:01.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:01.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:01.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:01.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:01.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:01.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:01.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:01.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:38.913151+0000 '>' 2026-02-20T16:42:38.913151+0000 2026-02-20T16:43:01.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:02.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:02.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:58.896677+0000 '>' 2026-02-20T16:42:38.913151+0000 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 227563"' 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 227563' 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:03.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 227564"' 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 227564' 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 227563 227564' 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 227563 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/227566: /' 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:43:03.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/227568: /' 2026-02-20T16:43:03.611 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.611 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:03.612 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: _ 2026-02-20T16:43:03.613 INFO:tasks.workunit.client.0.trial013.stderr:227566: hinfo_key 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: snapset 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:03.614 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:03.615 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: _ 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: hinfo_key 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: snapset 2026-02-20T16:43:03.616 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:03.617 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:03.618 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:03.619 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: start osd.5 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:03.620 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:03.621 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-20T16:43:03.622 INFO:tasks.workunit.client.0.trial013.stderr:227568: start osd.9 2026-02-20T16:43:05.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:05.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-20T16:43:05.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-20T16:43:05.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:05.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2026-02-20T16:43:03.625+0000 7f2a3dc5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2026-02-20T16:43:03.629+0000 7f2a3dc5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2026-02-20T16:43:03.629+0000 7f2a3dc5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:05.101 INFO:tasks.workunit.client.0.trial013.stderr:227568: 0 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2026-02-20T16:43:04.105+0000 7f2a3dc5ca40 -1 Falling back to public interface 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2026-02-20T16:43:04.809+0000 7f2a3dc5ca40 -1 osd.9 92 log_to_monitors true 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: 1 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-20T16:43:05.102 INFO:tasks.workunit.client.0.trial013.stderr:227568: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6842/2757487409,v1:127.0.0.1:6843/2757487409] [v2:127.0.0.1:6844/2757487409,v1:127.0.0.1:6845/2757487409] exists,up b3f2296f-f3e9-4f70-91f3-adb3ac619a32 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:05.103 INFO:tasks.workunit.client.0.trial013.stderr:227568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.svery_ops 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2026-02-20T16:43:03.625+0000 7f1d68d87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2026-02-20T16:43:03.629+0000 7f1d68d87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.104 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2026-02-20T16:43:03.629+0000 7f1d68d87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: 0 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2026-02-20T16:43:04.113+0000 7f1d68d87a40 -1 Falling back to public interface 2026-02-20T16:43:05.105 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2026-02-20T16:43:04.805+0000 7f1d68d87a40 -1 osd.5 92 log_to_monitors true 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: 1 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: osd.5 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [91,92) [v2:127.0.0.1:6874/1538224949,v1:127.0.0.1:6875/1538224949] [v2:127.0.0.1:6876/1538224949,v1:127.0.0.1:6877/1538224949] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:05.106 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:05.107 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:05.107 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:05.107 INFO:tasks.workunit.client.0.trial013.stderr:227566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:05.664 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 1 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 2 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 3 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 4 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 5 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 6 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 7 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 8 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: 9' 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.665 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672989 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672989 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989' 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509468 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509468 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468' 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.666 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:05.667 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776146 2026-02-20T16:43:05.667 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776146 2026-02-20T16:43:05.667 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146' 2026-02-20T16:43:05.667 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperh:1593: get_timeout_delays: echo true 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: 1 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: 2 2026-02-20T16:43:05.676 INFO:tasks.workunit.client.0.trial013.stderr:227568: 3 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 4 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 5 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 6 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 7 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 8 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: 9' 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:05.677 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672990 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672990 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990' 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509469 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509469 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469' 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776147 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776147 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147' 2026-02-20T16:43:05.678 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.100 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperss.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247835 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247835 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835 5-412316860418' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953497 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953497 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835 5-412316860418 6-137438953497' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789977 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789977 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835 5-412316860418 6-137438953497 7-158913789977' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626456 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626456 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835 5-412316860418 6-137438953497 7-158913789977 8-180388626456' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509468 3-292057776146 4-98784247835 5-412316860418 6-137438953497 7-158913789977 8-180388626456 9-412316860418' 2026-02-20T16:43:06.101 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247836 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247836 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836' 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860419 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860419 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836 5-412316860419' 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953498 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953498 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836 5-412316860419 6-137438953498' 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:06.111 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789978 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789978 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836 5-412316860419 6-137438953498 7-158913789978' 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626457 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626457 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836 5-412316860419 6-137438953498 7-158913789978 8-180388626457' 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860419 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860419 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-64424509469 3-292057776147 4-98784247836 5-412316860419 6-137438953498 7-158913789978 8-180388626457 9-412316860419' 2026-02-20T16:43:06.112 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/uubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-20T16:43:07.758 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.0 seq 21474836510 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836510 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836510 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672989 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:07.759 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672989 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672989 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.1 seq 42949672989 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672989 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509468 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509468 2026-02-20T16:43:07.760 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.761 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509468 2026-02-20T16:43:07.761 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2026-02-20T16:43:07.761 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.2 seq 64424509468 2026-02-20T16:43:07.761 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:07.761 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509470 -lt 64424509468 2026-02-20T16:43:07.772 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.0 seq 21474836511 2026-02-20T16:43:07.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836511 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836511 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672990 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672990 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672990 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.1 seq 42949672990 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672990 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509469 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509469 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509469 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2026-02-20T16:43:07.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.2 seq 64424509469 2026-02-20T16:43:07.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:07.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509470 -lt 64424509469 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flussh_pg_stats: for s in $seqs 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776146 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776146 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776146 2026-02-20T16:43:08.256 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776146' 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.3 seq 292057776146 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776147 -lt 292057776146 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247835 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247835 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247835 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247835' 2026-02-20T16:43:08.257 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.4 seq 98784247835 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247836 -lt 98784247835 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-412316860418 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-412316860418 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.5 seq 412316860418 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:08.258 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860418 2026-02-20T16:43:08.259 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.259 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953497 2026-02-20T16:43:08.259 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.259 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standh_pg_stats: for s in $seqs 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776147 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-292057776147 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776147 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 292057776147' 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.3 seq 292057776147 2026-02-20T16:43:08.275 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776147 -lt 292057776147 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247836 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247836 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247836 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247836' 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.4 seq 98784247836 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:08.276 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247836 -lt 98784247836 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-412316860419 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-412316860419 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860419 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.5 seq 412316860419 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860419 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953498 2026-02-20T16:43:08.277 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.278 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:08.747 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standaalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953497 2026-02-20T16:43:08.747 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.747 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953497 2026-02-20T16:43:08.747 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953497' 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.6 seq 137438953497 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953499 -lt 137438953497 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789977 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789977 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789977 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789977' 2026-02-20T16:43:08.748 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.7 seq 158913789977 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789978 -lt 158913789977 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626456 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626456 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626456 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626456' 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.8 seq 180388626456 2026-02-20T16:43:08.749 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626457 -lt 180388626456 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860418 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860418 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-20T16:43:08.773 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 slone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953498 2026-02-20T16:43:08.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953498 2026-02-20T16:43:08.773 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953498' 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.6 seq 137438953498 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953499 -lt 137438953498 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789978 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789978 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789978 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789978' 2026-02-20T16:43:08.774 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.7 seq 158913789978 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789978 -lt 158913789978 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626457 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626457 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626457 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626457' 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.8 seq 180388626457 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:08.775 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626457 -lt 180388626457 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860419 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860419 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:08.776 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860419 2026-02-20T16:43:09.708 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seeq 412316860418' 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: waiting osd.9 seq 412316860418 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860418 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:09.709 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-20T16:43:09.710 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:09.711 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/cloq 412316860419' 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: waiting osd.9 seq 412316860419 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860419 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:09.750 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:09.751 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:09.752 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:10.645 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clonne.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:10.645 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:10.645 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:10.645 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:10.646 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:10.646 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:43:10.646 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:10.646 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:10.646 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:10.647 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:10.648 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:10.648 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:10.648 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:10.648 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:10.648 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubune.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:43:10.724 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:10.725 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubunttu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:11.585 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:11.586 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:227566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:11.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:11.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:11.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 227564 2026-02-20T16:43:11.688 INFO:tasks.workunit.client.0.trial013.stderr:u/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:11.689 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:11.690 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:227568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:227568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:11.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:43:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:43:11.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:43:11.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-20T16:43:11.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-20T16:43:11.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-20T16:43:11.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 230826"' 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 230826' 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 230827"' 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 230827' 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 230826 230827' 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:43:11.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:11.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 230826 2026-02-20T16:43:11.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:43:11.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/230829: /' 2026-02-20T16:43:11.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:43:11.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/230831: /' 2026-02-20T16:43:12.362 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:43:12.363 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:43:12.364 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:12.365 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:12.366 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:12.366 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:12.366 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:43:12.366 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:12.367 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: start osd.3 2026-02-20T16:43:12.368 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:2230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:12.379 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:12.380 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:12.381 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:12.382 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:12.382 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:12.382 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:43:12.382 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:43:12.382 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:12.383 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: start osd.5 2026-02-20T16:43:12.384 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:13.856 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2026-02-20T16:43:12.389+0000 7fbb38488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2026-02-20T16:43:12.389+0000 7fbb38488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.857 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2026-02-20T16:43:12.389+0000 7fbb38488a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: 0 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:43:13.858 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2026-02-20T16:43:12.893+0000 7fbb38488a40 -1 Falling back to public interface 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2026-02-20T16:43:13.597+0000 7fbb38488a40 -1 osd.3 98 log_to_monitors true 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: 1 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:43:13.859 INFO:tasks.workunit.client.0.trial013.stderr:230829: osd.3 up in weight 1 up_from 101 up_thru 96 down_at 98 last_clean_interval [68,98) [v2:127.0.0.1:6826/638820642,v1:127.0.0.1:6827/638820642] [v2:127.0.0.1:6828/638820642,v1:127.0.0.1:6829/638820642] exists,up f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:13.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:13.869 INFO:tasks.workunit.client.0.trial013.stderr:230829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_30831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:43:13.869 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2026-02-20T16:43:12.397+0000 7f7abb58fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2026-02-20T16:43:12.397+0000 7f7abb58fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2026-02-20T16:43:12.397+0000 7f7abb58fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:43:13.870 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: 0 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2026-02-20T16:43:12.905+0000 7f7abb58fa40 -1 Falling back to public interface 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2026-02-20T16:43:13.609+0000 7f7abb58fa40 -1 osd.5 97 log_to_monitors true 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:13.871 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: 1 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: osd.5 up in weight 1 up_from 101 up_thru 96 down_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6874/1271776045,v1:127.0.0.1:6875/1271776045] [v2:127.0.0.1:6876/1271776045,v1:127.0.0.1:6877/1271776045] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:13.872 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:13.873 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:13.873 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:13.873 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:13.873 INFO:tasks.workunit.client.0.trial013.stderr:230831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: timeout_delays: echo true 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:14.415 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 1 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 2 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 3 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 4 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 5 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 6 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 7 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 8 2026-02-20T16:43:14.416 INFO:tasks.workunit.client.0.trial013.stderr:230829: 9' 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672993 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672993 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993' 2026-02-20T16:43:14.417 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509472 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509472 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472' 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898' 2026-02-20T16:43:14.418 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flget_timeout_delays: echo true 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:14.427 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 1 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 2 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 3 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 4 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 5 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 6 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 7 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 8 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: 9' 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:14.428 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836515 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836515 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515' 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672994 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672994 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994' 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:14.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509473 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509473 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473' 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899' 2026-02-20T16:43:14.430 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.848 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258ush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247839 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247839 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839' 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839 5-433791696898' 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.849 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953501 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953501 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839 5-433791696898 6-137438953501' 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789981 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789981 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839 5-433791696898 6-137438953501 7-158913789981' 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626460 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626460 2026-02-20T16:43:14.850 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839 5-433791696898 6-137438953501 7-158913789981 8-180388626460' 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860421 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860421 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-433791696898 4-98784247839 5-433791696898 6-137438953501 7-158913789981 8-180388626460 9-412316860421' 2026-02-20T16:43:14.851 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:14.860 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephte: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:14.860 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247840 2026-02-20T16:43:14.860 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247840 2026-02-20T16:43:14.860 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840' 2026-02-20T16:43:14.860 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840 5-433791696899' 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953502 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953502 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840 5-433791696899 6-137438953502' 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.861 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789982 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789982 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840 5-433791696899 6-137438953502 7-158913789982' 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626461 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626461 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840 5-433791696899 6-137438953502 7-158913789982 8-180388626461' 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860422 2026-02-20T16:43:14.862 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860422 2026-02-20T16:43:14.863 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-42949672994 2-64424509473 3-433791696899 4-98784247840 5-433791696899 6-137438953502 7-158913789982 8-180388626461 9-412316860422' 2026-02-20T16:43:14.863 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.0 seq 21474836514 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836514 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672993 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672993 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672993 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.1 seq 42949672993 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:15.343 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672993 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509472 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509472 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509472 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.2 seq 64424509472 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509473 -lt 64424509472 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696898 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:15.344 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696898 2026-02-20T16:43:15.372 INFO:tasks.workunit.client.0.trial013.stderr:230829phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836515 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836515 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836515 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.0 seq 21474836515 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836515 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672994 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:15.373 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672994 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672994 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.1 seq 42949672994 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672994 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509473 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509473 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509473 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.2 seq 64424509473 2026-02-20T16:43:15.374 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509473 -lt 64424509473 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696899 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:15.375 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696899 2026-02-20T16:43:15.841 INFO:tasks.workunit.client.0.trial013.stderr:23: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.841 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-20T16:43:15.841 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696898' 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.3 seq 433791696898 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247839 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247839 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.842 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247839 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247839' 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.4 seq 98784247839 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247840 -lt 98784247839 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-433791696898 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-433791696898 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 433791696898' 2026-02-20T16:43:15.843 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.5 seq 433791696898 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953501 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953501 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953501 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953501' 2026-02-20T16:43:15.844 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.6 seq 137438953501 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/0831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696899' 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.3 seq 433791696899 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:15.888 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247840 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247840 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247840 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247840' 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.4 seq 98784247840 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247840 -lt 98784247840 2026-02-20T16:43:15.889 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-433791696899 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-433791696899 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 433791696899' 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.5 seq 433791696899 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:15.890 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953502 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953502 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953502 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953502' 2026-02-20T16:43:15.891 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.6 seq 137438953502 2026-02-20T16:43:17.491 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubucephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:17.491 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953502 -lt 137438953501 2026-02-20T16:43:17.491 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.491 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789981 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789981 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789981 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789981' 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.7 seq 158913789981 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789982 -lt 158913789981 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.492 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626460 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626460 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626460 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626460' 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.8 seq 180388626460 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626459 -lt 180388626460 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:17.493 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626461 -lt 180388626460 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860421 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860421 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.494 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860421 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 41231686ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953502 -lt 137438953502 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789982 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789982 2026-02-20T16:43:17.552 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789982 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789982' 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.7 seq 158913789982 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789982 -lt 158913789982 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626461 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626461 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626461 2026-02-20T16:43:17.553 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626461' 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.8 seq 180388626461 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626459 -lt 180388626461 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626461 -lt 180388626461 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860422 2026-02-20T16:43:17.554 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:17.555 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:17.555 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860422 2026-02-20T16:43:17.555 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:17.555 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860422 2026-02-20T16:43:18.466 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 41230421' 2026-02-20T16:43:18.466 INFO:tasks.workunit.client.0.trial013.stderr:230829: waiting osd.9 seq 412316860421 2026-02-20T16:43:18.466 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:18.466 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860423 -lt 412316860421 2026-02-20T16:43:18.466 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:18.467 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:18.468 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:18.469 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:18.469 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:18.469 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:18.469 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:18.469 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:18.530 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.016860422' 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: waiting osd.9 seq 412316860422 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860423 -lt 412316860422 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:18.531 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-20T16:43:18.532 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:18.533 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:19.429 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.clie/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:19.429 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:19.429 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:19.429 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:19.429 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:19.430 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:19.431 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:230829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:19.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:19.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:19.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 230827 2026-02-20T16:43:19.484 INFO:tasks.workunit.client.0.trial013.stderr:nt.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:19.484 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:19.484 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:19.484 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T16:43:19.485 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:19.486 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:230831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:19.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:19.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:43:19.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T16:43:19.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:43:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-20T16:43:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-20T16:43:19.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:43:19.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:43:19.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:19.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:19.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:19.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:19.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:19.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:43:19.951 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:19.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:19.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:19.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:20.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:58.896677+0000 '>' 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:20.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:21.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:21.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:21.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:21.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:21.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:21.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:21.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:21.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:58.896677+0000 '>' 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:21.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:22.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:22.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:22.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:22.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:22.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:22.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:22.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:22.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:58.896677+0000 '>' 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:22.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:23.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:23.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:23.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:23.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:23.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:23.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:23.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:23.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:42:58.896677+0000 '>' 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:23.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:24.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:24.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:24.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:43:24.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:43:24.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:24.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:24.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:43:24.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:20.493326+0000 '>' 2026-02-20T16:42:58.896677+0000 2026-02-20T16:43:24.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:43:24.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-20T16:43:24.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 234482"' 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 234482' 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 234484"' 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 234484' 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 234482 234484' 2026-02-20T16:43:24.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T16:43:24.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:24.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 234482 2026-02-20T16:43:24.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/234485: /' 2026-02-20T16:43:24.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:24.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/234487: /' 2026-02-20T16:43:25.250 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.250 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-20T16:43:25.251 INFO:tasks.workunit.client.0.trial013.stderr:234485: _ 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: hinfo_key 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: snapset 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T16:43:25.252 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T16:43:25.253 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T16:43:25.254 INFO:tasks.workunit.client.0.trial013.stderr:234485: start osd.3 2026-02-20T16:43:25.271 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.271 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:25.271 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-20T16:43:25.271 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:25.271 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:25.272 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: _ 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: hinfo_key 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: snapset 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:25.273 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:25.274 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-20T16:43:25.274 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:25.274 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-20T16:43:25.274 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:25.274 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:25.275 INFO:tasks.workunit.client.0.trial013.stderr:234487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:25.276 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:25.277 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:25.277 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-20T16:43:25.277 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-20T16:43:25.277 INFO:tasks.workunit.client.0.trial013.stderr:234487: start osd.5 2026-02-20T16:43:26.739 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recolog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:25.269+0000 7fcdb691ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:25.269+0000 7fcdb691ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.740 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:25.269+0000 7fcdb691ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: 0 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:25.757+0000 7fcdb691ca40 -1 Falling back to public interface 2026-02-20T16:43:26.741 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:25.757+0000 7fcdb58dc640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6826/0: (98) Address already in use 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2026-02-20T16:43:26.421+0000 7fcdb691ca40 -1 osd.3 102 log_to_monitors true 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: 1 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: osd.3 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6827/60666689,v1:127.0.0.1:6829/60666689] [v2:127.0.0.1:6831/60666689,v1:127.0.0.1:6833/60666689] exists,up f51cf9ba-a841-4d1d-b4c3-55aa0e231d25 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:26.742 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:26.743 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: lvery_ops 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:26.762 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2026-02-20T16:43:25.297+0000 7f3d369a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2026-02-20T16:43:25.297+0000 7f3d369a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2026-02-20T16:43:25.301+0000 7f3d369a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: 0 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:26.763 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2026-02-20T16:43:25.757+0000 7f3d369a8a40 -1 Falling back to public interface 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2026-02-20T16:43:26.397+0000 7f3d369a8a40 -1 osd.5 102 log_to_monitors true 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: 1 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6826/3487430627,v1:127.0.0.1:6828/3487430627] [v2:127.0.0.1:6830/3487430627,v1:127.0.0.1:6832/3487430627] exists,up 59df7655-1e17-4ef4-86c6-42ed9af0be0a 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:26.764 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:26.765 INFO:tasks.workunit.client.0.trial013.stderr:234487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heocal cur_active_clean 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:27.298 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 1 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 2 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 3 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 4 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 5 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 6 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 7 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 8 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: 9' 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997' 2026-02-20T16:43:27.299 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509477 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509477 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477' 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-20T16:43:27.300 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:23448lpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:27.320 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 1 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 2 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 3 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 4 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 5 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 6 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 7 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 8 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: 9' 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:27.321 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836519 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836519 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519' 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672998 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672998 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998' 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509478 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509478 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478' 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-20T16:43:27.322 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379' 2026-02-20T16:43:27.323 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.730 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-5: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378' 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247843 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247843 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843' 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-20T16:43:27.731 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843 5-455266533378' 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953506 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953506 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843 5-455266533378 6-137438953506' 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789985 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789985 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843 5-455266533378 6-137438953506 7-158913789985' 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:27.732 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626464 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626464 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843 5-455266533378 6-137438953506 7-158913789985 8-180388626464' 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860426 2026-02-20T16:43:27.733 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860426 2026-02-20T16:43:27.755 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.chelpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247844 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247844 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844' 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-20T16:43:27.756 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844 5-455266533379' 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953507 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953507 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844 5-455266533379 6-137438953507' 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789986 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789986 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844 5-455266533379 6-137438953507 7-158913789986' 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.757 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626465 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626465 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844 5-455266533379 6-137438953507 7-158913789986 8-180388626465' 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860427 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860427 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-455266533379 4-98784247844 5-455266533379 6-137438953507 7-158913789986 8-180388626465 9-412316860427' 2026-02-20T16:43:27.758 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234487: /lient.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-455266533378 4-98784247843 5-455266533378 6-137438953506 7-158913789985 8-180388626464 9-412316860426' 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.242 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.0 seq 21474836518 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836518 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.243 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.1 seq 42949672997 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672999 -lt 42949672997 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.244 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509477 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509477 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509477 2026-02-20T16:43:29.245 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.2 seq 64424509/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836519 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836519 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836519 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.0 seq 21474836519 2026-02-20T16:43:29.405 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836519 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836519 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672998 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672998 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672998 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.1 seq 42949672998 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672999 -lt 42949672998 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509478 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509478 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509478 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.2 seq 64424509478 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:29.406 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509478 -lt 64424509478 2026-02-20T16:43:29.904 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226477 2026-02-20T16:43:29.904 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:43:29.904 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509478 -lt 64424509477 2026-02-20T16:43:29.904 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533378 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533378 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533378' 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.3 seq 455266533378 2026-02-20T16:43:29.905 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247843 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247843 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247843 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247843' 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.4 seq 98784247843 2026-02-20T16:43:29.906 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247845 -lt 98784247843 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533378 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533378 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533378' 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.5 seq 455266533378 2026-02-20T16:43:29.907 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:29.908 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-20T16:43:29.908 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.908 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/q6: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.908 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533379 2026-02-20T16:43:29.908 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533379 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533379' 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.3 seq 455266533379 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.909 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247844 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-98784247844 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247844 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 98784247844' 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.4 seq 98784247844 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247845 -lt 98784247844 2026-02-20T16:43:29.910 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533379 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533379 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.5 seq 455266533379 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-20T16:43:29.911 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:29.912 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953507 2026-02-20T16:43:29.912 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:29.912 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qaa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953506 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953506 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953506 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953506' 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.6 seq 137438953506 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953507 -lt 137438953506 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789985 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789985 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.395 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789985 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789985' 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.7 seq 158913789985 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789986 -lt 158913789985 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626464 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626464 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626464 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626464' 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.8 seq 180388626464 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626466 -lt 180388626464 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860426 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:30.396 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860426 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:2344/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953507 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953507 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953507' 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.6 seq 137438953507 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953507 -lt 137438953507 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789986 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789986 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789986 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789986' 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.7 seq 158913789986 2026-02-20T16:43:30.411 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789986 -lt 158913789986 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626465 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626465 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626465 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626465' 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.8 seq 180388626465 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626466 -lt 180388626465 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860427 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860427 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:30.412 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860427 2026-02-20T16:43:31.091 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting o85: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860426 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 412316860426' 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: waiting osd.9 seq 412316860426 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860427 -lt 412316860426 2026-02-20T16:43:31.093 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:31.094 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:234485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T16:43:31.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 234484 2026-02-20T16:43:31.137 INFO:tasks.workunit.client.0.trial013.stderr:sd.9 seq 412316860427' 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: waiting osd.9 seq 412316860427 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860427 -lt 412316860427 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:31.138 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:234487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T16:43:31.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T16:43:31.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T16:43:31.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T16:43:31.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-20T16:43:31.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-20T16:43:31.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:43:31.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:43:31.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:43:31.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:43:31.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:43:31.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:31.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:31.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:31.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:31.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:31.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:31.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:43:31.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:43:31.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:43:31.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:43:31.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:43:31.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:43:31.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:43:31.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:43:31.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:43:31.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:43:31.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:43:31.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:43:31.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:43:31.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:43:31.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:43:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:43:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:43:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:43:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:43:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:31.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:31.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:43:31.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:43:31.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:43:31.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:43:31.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:43:31.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:43:31.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:43:31.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:43:31.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:43:31.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:43:31.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:43:31.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:43:31.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:43:31.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:43:31.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:43:31.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:43:31.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:43:31.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:43:31.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:43:31.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:43:31.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-20T16:43:31.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:43:31.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:43:31.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:43:31.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:43:31.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.565 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:31.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T16:43:31.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:43:31.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:43:31.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:43:31.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:43:31.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:43:31.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:43:31.598 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:43:31.598 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:43:31.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:43:31.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:43:31.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:43:31.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:43:31.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:43:31.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:43:31.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.655 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.656 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:43:31.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:43:31.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:43:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:43:31.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:43:31.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:31.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:31.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:31.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.809 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:31.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:43:31.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:43:31.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:31.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:43:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:43:31.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:43:31.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=859c08cd-c4c6-448d-8017-eeda92c2dd5d 2026-02-20T16:43:31.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 859c08cd-c4c6-448d-8017-eeda92c2dd5d' 2026-02-20T16:43:31.831 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 859c08cd-c4c6-448d-8017-eeda92c2dd5d 2026-02-20T16:43:31.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:43:31.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzj5hpn9hJMhAAI3XCQ27/dINC8SienELNIw== 2026-02-20T16:43:31.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzj5hpn9hJMhAAI3XCQ27/dINC8SienELNIw=="}' 2026-02-20T16:43:31.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 859c08cd-c4c6-448d-8017-eeda92c2dd5d -i td/osd-scrub-repair/0/new.json 2026-02-20T16:43:31.933 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:43:31.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:43:31.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzj5hpn9hJMhAAI3XCQ27/dINC8SienELNIw== --osd-uuid 859c08cd-c4c6-448d-8017-eeda92c2dd5d 2026-02-20T16:43:31.957 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:31.953+0000 7eff536b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:31.959 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:31.957+0000 7eff536b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:31.960 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:31.957+0000 7eff536b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:31.960 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:31.957+0000 7eff536b1a40 -1 bdev(0x55e035a5ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:43:31.960 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:31.957+0000 7eff536b1a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:43:32.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:43:32.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:43:32.673 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:43:32.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:43:32.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:43:32.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:43:32.770 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:43:32.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:32.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:43:32.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:43:32.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:43:32.785 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:32.781+0000 7fed06be9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:32.786 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:32.781+0000 7fed06be9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:32.787 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:32.785+0000 7fed06be9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:32.865 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:43:32.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:32.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:32.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:43:32.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:33.212 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:33.209+0000 7fed06be9a40 -1 Falling back to public interface 2026-02-20T16:43:33.866 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:33.861+0000 7fed06be9a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:43:33.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:33.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:33.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:33.960 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:43:33.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:33.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:43:34.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:35.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:35.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:35.083 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:43:35.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:43:35.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:35.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:43:35.240 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2049247855,v1:127.0.0.1:6801/2049247855] [v2:127.0.0.1:6802/2049247855,v1:127.0.0.1:6803/2049247855] exists,up 859c08cd-c4c6-448d-8017-eeda92c2dd5d 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:43:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:35.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:35.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:35.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:35.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:43:35.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:43:35.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:43:35.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:43:35.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:43:35.248 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:43:35.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a541a422-8b1a-4cb0-8d99-003f72087442' 2026-02-20T16:43:35.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:43:35.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3j5hpWdxhDxAAth3m08Y1ct8Pjqmg7kdxaw== 2026-02-20T16:43:35.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3j5hpWdxhDxAAth3m08Y1ct8Pjqmg7kdxaw=="}' 2026-02-20T16:43:35.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a541a422-8b1a-4cb0-8d99-003f72087442 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:43:35.422 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:43:35.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:43:35.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3j5hpWdxhDxAAth3m08Y1ct8Pjqmg7kdxaw== --osd-uuid a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:43:35.449 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:35.445+0000 7f6c5804ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:35.451 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:35.449+0000 7f6c5804ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:35.452 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:35.449+0000 7f6c5804ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:35.452 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:35.449+0000 7f6c5804ea40 -1 bdev(0x5605cb1b6c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:43:35.452 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:35.449+0000 7f6c5804ea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:43:36.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:43:36.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:43:36.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:43:36.120 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:43:36.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:43:36.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:43:36.335 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:43:36.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:36.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:43:36.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:43:36.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:43:36.358 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:36.353+0000 7f0100d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:36.360 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:36.357+0000 7f0100d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:36.361 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:36.357+0000 7f0100d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:36.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:43:36.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:36.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:43:36.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:36.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:36.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:36.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:36.493 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:43:36.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:36.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:43:36.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:36.780 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:36.777+0000 7f0100d3da40 -1 Falling back to public interface 2026-02-20T16:43:37.442 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:37.437+0000 7f0100d3da40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:43:37.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:37.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:37.654 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:43:37.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:37.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:37.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:43:37.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:38.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:38.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:38.812 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:43:38.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:43:38.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:38.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:43:38.974 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/630561243,v1:127.0.0.1:6811/630561243] [v2:127.0.0.1:6812/630561243,v1:127.0.0.1:6813/630561243] exists,up a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:43:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2026-02-20T16:43:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:43:39.125 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:43:39.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:43:39.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:43:39.342 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:43:39.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:43:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:43:40.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2026-02-20T16:43:40.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:40.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:40.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:40.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:40.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:40.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:40.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:40.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:40.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:40.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:40.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:40.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:40.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:40.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:40.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:40.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:40.905 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:43:40.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:40.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:40.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:43:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:43:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:43:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:40.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:41.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-20T16:43:41.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-20T16:43:41.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-20T16:43:41.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:41.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:43:41.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:41.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:41.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:43:41.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:41.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:43:41.058 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:43:41.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:43:41.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:41.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T16:43:41.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:42.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:42.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:42.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:43:42.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:42.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-20T16:43:42.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:42.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:42.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-20T16:43:42.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:42.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-20T16:43:42.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-20T16:43:42.389 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705666 2026-02-20T16:43:42.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:42.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-20T16:43:42.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:42.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:42.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:42.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:43:42.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:42.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:42.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:42.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:42.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:42.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:43.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:43:43.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:43:43.305 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:43:43.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:43:43.511 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:43:43.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:43:43.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:43:43.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:43:43.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2026-02-20T16:43:43.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-20T16:43:43.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:43:43.550 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-20T16:43:43.550 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-20T16:43:43.550 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:43:43.551 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:43:43.551 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:43:43.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:43:43.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:43:43.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:43:43.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:43.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:43:44.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T16:43:44.375 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:44.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:44.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:44.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:44.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:43:44.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:43:44.475 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:43:44.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:44.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:43:44.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:43:44.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:44.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:44.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:44.487 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:44.485+0000 7fbd8d9aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:44.489 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:44.485+0000 7fbd8d9aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:44.489 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:44.485+0000 7fbd8d9aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:43:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:43:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:44.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:44.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:44.627 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:43:44.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:44.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:43:44.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:44.941 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:44.937+0000 7fbd8d9aea40 -1 Falling back to public interface 2026-02-20T16:43:45.580 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:45.577+0000 7fbd8d9aea40 -1 osd.0 18 log_to_monitors true 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:45.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:43:45.949 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6800/2505813557,v1:127.0.0.1:6801/2505813557] [v2:127.0.0.1:6802/2505813557,v1:127.0.0.1:6803/2505813557] exists,up 859c08cd-c4c6-448d-8017-eeda92c2dd5d 2026-02-20T16:43:45.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:45.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:45.951 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:45.951 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:45.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:45.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:45.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:46.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:46.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:46.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:46.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:46.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:46.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:46.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:46.217 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:43:46.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:46.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:46.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-20T16:43:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-20T16:43:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-20T16:43:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:46.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:46.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-20T16:43:46.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-20T16:43:46.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705668' 2026-02-20T16:43:46.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:46.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-20T16:43:46.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:46.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:46.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-20T16:43:46.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:46.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-20T16:43:46.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-20T16:43:46.368 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280514 2026-02-20T16:43:46.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:46.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-20T16:43:46.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:47.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:47.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:47.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-20T16:43:47.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:48.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:43:48.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:48.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-20T16:43:48.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:48.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-20T16:43:48.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:48.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:48.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-20T16:43:48.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:48.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-20T16:43:48.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-20T16:43:48.861 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705668 2026-02-20T16:43:48.861 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:49.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-20T16:43:49.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:49.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:49.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:49.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:43:49.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:49.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:49.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:49.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:49.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:49.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:49.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:49.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:43:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:49.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-20T16:43:49.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-20T16:43:49.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:43:49.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-20T16:43:49.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:49.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:49.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:49.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:49.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-20T16:43:50.021 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-20T16:43:50.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:50.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-20T16:43:50.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:50.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:43:50.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:43:50.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:50.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:43:50.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:50.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:50.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:50.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:50.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:39.342199+0000 '>' 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:50.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:51.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:51.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:51.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:43:51.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:51.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:51.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:51.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:51.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:39.342199+0000 '>' 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:51.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:52.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:52.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:52.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:43:52.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:52.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:52.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:52.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:52.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:39.342199+0000 '>' 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:52.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:53.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:53.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:43:53.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:53.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:53.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:53.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:53.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:39.342199+0000 '>' 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:53.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:43:54.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:43:54.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:43:54.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:43:54.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:43:54.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:43:54.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:43:54.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:43:54.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:50.398012+0000 '>' 2026-02-20T16:43:39.342199+0000 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:54.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:54.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:54.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:55.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:43:55.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:43:55.217 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:43:55.217 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:43:55.319 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:43:55.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:43:55.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:43:55.321 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:43:55.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:43:55.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:43:55.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:43:55.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:43:55.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:43:55.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:43:55.337 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:55.332+0000 7f733d3baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:55.338 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:55.332+0000 7f733d3baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:55.339 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:55.336+0000 7f733d3baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:43:55.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:43:55.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:55.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:43:55.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:43:55.741 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:55.736+0000 7f733d3baa40 -1 Falling back to public interface 2026-02-20T16:43:56.415 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:43:56.412+0000 7f733d3baa40 -1 osd.1 26 log_to_monitors true 2026-02-20T16:43:56.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:43:56.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:43:56.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:43:56.647 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:43:56.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:43:56.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [9,26) [v2:127.0.0.1:6810/4239654798,v1:127.0.0.1:6811/4239654798] [v2:127.0.0.1:6812/4239654798,v1:127.0.0.1:6813/4239654798] exists,up a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:43:56.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:43:56.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:43:56.810 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:43:56.810 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:43:56.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:43:56.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:43:56.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:43:56.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:43:56.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:43:56.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:43:56.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:43:56.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:43:56.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:43:57.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:43:57.082 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:43:57.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:43:57.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:57.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:43:57.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280517 2026-02-20T16:43:57.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280517 2026-02-20T16:43:57.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517' 2026-02-20T16:43:57.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:43:57.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:43:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-20T16:43:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-20T16:43:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517 1-124554051586' 2026-02-20T16:43:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:57.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280517 2026-02-20T16:43:57.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:57.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:43:57.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280517 2026-02-20T16:43:57.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:57.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280517 2026-02-20T16:43:57.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2026-02-20T16:43:57.234 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280517 2026-02-20T16:43:57.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280516 -lt 94489280517 2026-02-20T16:43:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:43:58.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:43:58.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:43:58.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280517 2026-02-20T16:43:58.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:43:58.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051586 2026-02-20T16:43:58.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:43:58.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:43:58.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051586 2026-02-20T16:43:58.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:43:58.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-20T16:43:58.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2026-02-20T16:43:58.569 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 124554051586 2026-02-20T16:43:58.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:43:58.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-20T16:43:58.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:43:58.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:58.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:58.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:43:58.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:43:59.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:43:59.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:43:59.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:43:59.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:43:59.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:43:59.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:43:59.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:43:59.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:43:59.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:43:59.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2026-02-20T16:43:59.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-20T16:43:59.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:43:59.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:43:59.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:43:59.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2026-02-20T16:43:59.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:43:59.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:43:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:43:59.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:43:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-20T16:43:59.930 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:00.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:00.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:44:00.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:44:00.018 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:44:00.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:00.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:44:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:44:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:00.037 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:00.032+0000 7f6f85a07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:00.039 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:00.036+0000 7f6f85a07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:00.040 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:00.036+0000 7f6f85a07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:00.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:00.472 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:00.468+0000 7f6f85a07a40 -1 Falling back to public interface 2026-02-20T16:44:01.177 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:01.172+0000 7f6f85a07a40 -1 osd.1 30 log_to_monitors true 2026-02-20T16:44:01.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:01.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:01.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:01.340 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:01.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:01.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:01.500 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6810/3138446124,v1:127.0.0.1:6811/3138446124] [v2:127.0.0.1:6812/3138446124,v1:127.0.0.1:6813/3138446124] exists,up a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:01.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:01.502 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:01.502 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:01.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:01.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:01.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:01.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:01.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:01.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:01.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:01.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:01.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:01.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:01.771 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:44:01.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:01.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:01.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:01.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280519 2026-02-20T16:44:01.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280519 2026-02-20T16:44:01.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519' 2026-02-20T16:44:01.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:01.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:01.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-20T16:44:01.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-20T16:44:01.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519 1-146028888066' 2026-02-20T16:44:01.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:01.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280519 2026-02-20T16:44:01.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:01.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:01.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280519 2026-02-20T16:44:01.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:01.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280519 2026-02-20T16:44:01.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2026-02-20T16:44:01.925 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280519 2026-02-20T16:44:01.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:02.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280519 2026-02-20T16:44:02.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:03.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:03.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:03.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280519 -lt 94489280519 2026-02-20T16:44:03.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:03.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888066 2026-02-20T16:44:03.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:03.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:03.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888066 2026-02-20T16:44:03.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:03.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-20T16:44:03.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2026-02-20T16:44:03.258 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 146028888066 2026-02-20T16:44:03.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:03.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-20T16:44:03.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:03.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:03.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:03.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:03.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:44:03.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:03.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:03.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-20T16:44:03.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:44:04.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:04.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:04.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:04.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-20T16:44:04.426 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:44:04.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:44:04.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:44:04.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:04.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:04.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:04.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:04.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:50.398012+0000 '>' 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:04.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:44:05.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:44:05.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:44:05.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:44:05.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:05.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:05.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:05.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:50.398012+0000 '>' 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:44:06.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:44:06.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:44:06.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:44:06.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:06.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:06.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:06.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:06.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:50.398012+0000 '>' 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:06.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:44:07.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:44:07.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:44:07.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:44:07.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:07.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:07.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:07.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:08.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:43:50.398012+0000 '>' 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:08.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:44:09.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:44:09.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:44:09.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:44:09.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:44:09.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:44:09.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:44:09.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:44:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:05.223977+0000 '>' 2026-02-20T16:43:50.398012+0000 2026-02-20T16:44:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:44:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:44:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:09.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:09.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:09.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:09.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-20T16:44:09.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:09.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:09.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:44:09.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:09.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:09.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-20T16:44:09.602 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-20T16:44:09.602 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-20T16:44:09.602 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:44:09.603 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:44:09.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:44:09.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:09.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:09.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:44:09.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:09.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:09.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:09.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:09.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:44:09.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:09.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:09.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:09.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:09.716+0000 7f8ae1570a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:09.720 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:09.716+0000 7f8ae1570a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:09.721 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:09.716+0000 7f8ae1570a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:09.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:09.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:10.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:10.124 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:10.120+0000 7f8ae1570a40 -1 Falling back to public interface 2026-02-20T16:44:10.777 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:10.772+0000 7f8ae1570a40 -1 osd.1 36 log_to_monitors true 2026-02-20T16:44:11.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:11.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:11.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:11.027 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:11.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:11.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [34,36) [v2:127.0.0.1:6810/47599469,v1:127.0.0.1:6811/47599469] [v2:127.0.0.1:6812/47599469,v1:127.0.0.1:6813/47599469] exists,up a541a422-8b1a-4cb0-8d99-003f72087442 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:11.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:11.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:11.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:11.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:11.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:11.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:11.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:11.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:11.458 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:44:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:11.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:11.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280522 2026-02-20T16:44:11.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280522 2026-02-20T16:44:11.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280522' 2026-02-20T16:44:11.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:11.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:11.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-20T16:44:11.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-20T16:44:11.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280522 1-171798691842' 2026-02-20T16:44:11.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:11.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280522 2026-02-20T16:44:11.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:11.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:11.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280522 2026-02-20T16:44:11.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:11.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280522 2026-02-20T16:44:11.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280522' 2026-02-20T16:44:11.608 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 94489280522 2026-02-20T16:44:11.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:11.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280522 2026-02-20T16:44:11.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:12.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:12.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280522 -lt 94489280522 2026-02-20T16:44:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:12.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691842 2026-02-20T16:44:12.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:12.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:12.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691842 2026-02-20T16:44:12.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:12.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-20T16:44:12.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2026-02-20T16:44:12.934 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 171798691842 2026-02-20T16:44:12.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:13.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-20T16:44:13.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:13.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:13.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:13.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:44:13.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:13.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:44:13.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:13.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:13.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:44:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:13.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:13.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:44:13.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:13.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:13.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:13.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:13.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:44:13.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:44:13.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:44:13.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:44:13.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:44:13.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:44:13.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:44:13.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:44:13.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:44:13.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:44:13.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:44:13.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:44:13.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:44:13.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:44:13.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:13.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:13.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:13.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:44:13.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:44:13.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:44:13.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:44:13.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:44:13.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:44:13.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:44:13.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:44:13.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:44:13.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:44:13.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:44:13.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:44:13.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:44:13.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:44:13.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:44:13.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:44:13.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:44:13.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:44:13.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:44:13.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:44:13.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:44:13.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:44:13.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=false 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:44:13.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:44:13.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:44:13.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:13.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:44:13.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:44:13.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:44:13.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:44:13.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:44:13.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:44:13.871 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.872 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.872 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:44:13.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:44:13.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:44:13.929 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:44:13.930 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:44:13.930 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:13.931 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:13.931 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:44:13.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:44:13.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:44:13.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:44:13.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:44:14.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:44:14.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:14.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:14.079 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:14.079 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:14.079 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:14.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:14.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:44:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:44:14.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:44:14.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:14.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:14.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:14.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:44:14.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:44:14.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:44:14.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:44:14.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:14.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6ba72c4e-583b-4176-8bf9-bc66306391c7' 2026-02-20T16:44:14.107 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:14.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:44:14.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBej5hpmlPyBhAAMsSuYnO0FVrpiJv9BhXCVA== 2026-02-20T16:44:14.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBej5hpmlPyBhAAMsSuYnO0FVrpiJv9BhXCVA=="}' 2026-02-20T16:44:14.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ba72c4e-583b-4176-8bf9-bc66306391c7 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:44:14.206 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:44:14.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:44:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBej5hpmlPyBhAAMsSuYnO0FVrpiJv9BhXCVA== --osd-uuid 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:14.231 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:14.228+0000 7fa7ceef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:14.233 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:14.228+0000 7fa7ceef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:14.234 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:14.228+0000 7fa7ceef2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:14.234 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:14.228+0000 7fa7ceef2a40 -1 bdev(0x56299d9cac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:44:14.235 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:14.228+0000 7fa7ceef2a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:44:15.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:44:15.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:44:15.016 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:44:15.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:44:15.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:44:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:44:15.121 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:44:15.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:15.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:44:15.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:44:15.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:44:15.145 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:15.140+0000 7f47ad7b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:15.147 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:15.144+0000 7f47ad7b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:15.148 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:15.144+0000 7f47ad7b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:15.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:44:15.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:15.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:44:15.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:15.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:15.216 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:44:15.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:15.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:15.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:15.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:15.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:15.549 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:15.544+0000 7f47ad7b1a40 -1 Falling back to public interface 2026-02-20T16:44:16.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:16.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:16.308 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:44:16.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:16.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:16.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:16.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:16.574 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:16.568+0000 7f47ad7b1a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:44:17.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:17.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:17.406 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:44:17.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:44:17.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:17.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:17.565 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3917815910,v1:127.0.0.1:6801/3917815910] [v2:127.0.0.1:6802/3917815910,v1:127.0.0.1:6803/3917815910] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:17.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:17.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:44:17.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:44:17.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:44:17.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:17.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 94229b16-0503-4483-afd2-4a432e39f2ed' 2026-02-20T16:44:17.570 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:17.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:44:17.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhj5hpUJG6IhAAU5TUYSoIx2Jgiqu49XiKZg== 2026-02-20T16:44:17.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhj5hpUJG6IhAAU5TUYSoIx2Jgiqu49XiKZg=="}' 2026-02-20T16:44:17.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 94229b16-0503-4483-afd2-4a432e39f2ed -i td/osd-scrub-repair/1/new.json 2026-02-20T16:44:17.746 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:44:17.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:44:17.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhj5hpUJG6IhAAU5TUYSoIx2Jgiqu49XiKZg== --osd-uuid 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:17.773 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:17.768+0000 7fc9a740ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:17.775 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:17.772+0000 7fc9a740ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:17.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:17.772+0000 7fc9a740ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:17.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:17.772+0000 7fc9a740ea40 -1 bdev(0x563c501d4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:44:17.777 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:17.772+0000 7fc9a740ea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:44:18.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:44:18.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:44:18.471 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:44:18.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:44:18.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:44:18.672 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:44:18.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:44:18.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:18.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:44:18.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:44:18.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:44:18.688 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:18.684+0000 7febe1290a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:18.690 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:18.684+0000 7febe1290a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:18.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:18.688+0000 7febe1290a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:18.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:18.833 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:44:18.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:18.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:18.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:19.056 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:19.052+0000 7febe1290a40 -1 Falling back to public interface 2026-02-20T16:44:19.707 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:19.704+0000 7febe1290a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:44:19.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:19.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:19.995 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:44:19.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:19.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:19.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:20.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:21.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:21.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:21.158 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:44:21.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:44:21.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:21.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:21.319 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/269830126,v1:127.0.0.1:6811/269830126] [v2:127.0.0.1:6812/269830126,v1:127.0.0.1:6813/269830126] exists,up 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:21.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:21.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:21.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:44:21.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:44:21.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:44:21.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:44:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:44:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3' 2026-02-20T16:44:21.327 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:44:21.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:44:21.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlj5hpWO0fFBAApgUnt7zQlw61sERR8FhiaA== 2026-02-20T16:44:21.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlj5hpWO0fFBAApgUnt7zQlw61sERR8FhiaA=="}' 2026-02-20T16:44:21.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:44:21.502 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:44:21.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:44:21.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlj5hpWO0fFBAApgUnt7zQlw61sERR8FhiaA== --osd-uuid 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:44:21.528 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:21.524+0000 7efc206eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:21.530 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:21.524+0000 7efc206eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:21.531 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:21.528+0000 7efc206eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:21.531 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:21.528+0000 7efc206eda40 -1 bdev(0x55c4baeecc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:44:21.531 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:21.528+0000 7efc206eda40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:44:22.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:44:22.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:44:22.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:44:22.164 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:44:22.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:44:22.369 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:44:22.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:44:22.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:22.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:44:22.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:44:22.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:44:22.394 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:22.388+0000 7fd53a36ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:22.395 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:22.392+0000 7fd53a36ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:22.396 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:22.392+0000 7fd53a36ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:22.529 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:44:22.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:22.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:22.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:44:22.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:22.784 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:22.780+0000 7fd53a36ba40 -1 Falling back to public interface 2026-02-20T16:44:23.345 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:23.340+0000 7fd53a36ba40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:44:23.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:23.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:23.692 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:44:23.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:23.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:23.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:44:23.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:24.859 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:44:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:44:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:44:25.020 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2658912007,v1:127.0.0.1:6819/2658912007] [v2:127.0.0.1:6820/2658912007,v1:127.0.0.1:6821/2658912007] exists,up 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:44:25.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:25.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:25.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:25.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-20T16:44:25.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:44:25.169 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:44:25.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:44:25.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:44:25.365 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:44:25.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:44:26.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:44:26.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-20T16:44:26.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-20T16:44:26.855 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T16:44:26.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:44:27.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-20T16:44:28.076 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:44:28.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:44:28.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:44:28.384 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:44:28.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:44:29.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:44:29.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:44:29.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:29.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:29.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:29.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:29.400 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:29.400 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:29.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:29.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:29.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:29.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:29.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:29.660 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:29.660 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:29.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:29.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T16:44:29.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T16:44:29.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T16:44:29.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:29.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:29.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:44:29.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:44:29.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-20T16:44:29.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:29.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:29.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-20T16:44:29.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-20T16:44:29.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-20T16:44:29.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:29.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T16:44:29.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:29.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:29.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T16:44:29.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:29.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T16:44:29.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T16:44:29.889 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T16:44:29.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:30.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:44:30.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:31.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:31.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:31.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T16:44:31.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:31.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:44:31.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:31.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:31.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:44:31.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:31.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:44:31.222 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:44:31.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:44:31.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:31.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T16:44:31.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:31.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-20T16:44:31.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:31.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:31.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-20T16:44:31.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:31.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-20T16:44:31.388 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509443 2026-02-20T16:44:31.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-20T16:44:31.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:31.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-20T16:44:31.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:31.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:31.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:31.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:31.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:31.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:31.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:31.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:31.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:31.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:32.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:32.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:32.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:32.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:44:32.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:32.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:32.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:32.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:32.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:32.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:32.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:32.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:32.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:32.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:32.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:32.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:32.363 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:32.363 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:32.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:32.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:32.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:32.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:44:32.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:44:32.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:44:32.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:32.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:32.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-20T16:44:32.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-20T16:44:32.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-20T16:44:32.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:32.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:32.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-20T16:44:32.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-20T16:44:32.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444' 2026-02-20T16:44:32.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:44:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:32.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:32.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:44:32.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:32.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:44:32.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:44:32.586 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:44:32.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:32.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:44:32.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:33.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:33.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:33.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:44:33.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:34.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:44:34.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:35.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-20T16:44:35.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:35.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-20T16:44:35.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:35.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:35.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-20T16:44:35.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:35.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-20T16:44:35.076 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672965 2026-02-20T16:44:35.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-20T16:44:35.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:35.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-20T16:44:35.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:35.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-20T16:44:35.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:35.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:35.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-20T16:44:35.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:35.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-20T16:44:35.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-20T16:44:35.244 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509444 2026-02-20T16:44:35.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:35.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-20T16:44:35.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:35.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:35.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:35.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:35.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:35.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:35.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:35.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:35.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:35.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:35.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:35.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:35.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:35.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-20T16:44:35.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:44:35.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-20T16:44:35.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:44:35.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:44:36.136 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:44:36.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:44:36.341 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:44:36.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:44:36.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:44:36.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:44:36.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-20T16:44:36.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:36.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:36.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:36.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:36.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:36.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:36.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:36.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:36.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:37.006 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:37.000+0000 7f7e4e6bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:37.007 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:37.004+0000 7f7e4e6bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:37.008 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:37.004+0000 7f7e4e6bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:37.147 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:37.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:37.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:37.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:37.372 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:37.368+0000 7f7e4e6bea40 -1 Falling back to public interface 2026-02-20T16:44:38.002 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:37.996+0000 7f7e4e6bea40 -1 osd.1 34 log_to_monitors true 2026-02-20T16:44:38.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:38.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:38.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:38.309 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:38.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:38.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:38.463 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/44738697,v1:127.0.0.1:6811/44738697] [v2:127.0.0.1:6812/44738697,v1:127.0.0.1:6813/44738697] exists,up 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:38.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:38.465 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:38.466 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:38.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:38.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:38.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:38.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:38.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:38.746 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:38.746 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:38.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:38.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:38.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:38.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:44:38.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:44:38.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:44:38.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:38.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-20T16:44:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-20T16:44:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-163208757250' 2026-02-20T16:44:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:38.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:38.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509446 2026-02-20T16:44:38.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509446 2026-02-20T16:44:38.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-163208757250 2-64424509446' 2026-02-20T16:44:38.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:38.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:44:38.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:38.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:38.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:44:38.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:38.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:44:38.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:44:38.966 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836488 2026-02-20T16:44:38.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:39.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:44:39.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:40.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:40.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:40.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:44:40.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:41.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:44:41.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:41.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:44:41.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:41.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-163208757250 2026-02-20T16:44:41.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:41.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:41.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-163208757250 2026-02-20T16:44:41.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:41.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-20T16:44:41.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2026-02-20T16:44:41.466 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 163208757250 2026-02-20T16:44:41.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:41.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-20T16:44:41.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:41.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509446 2026-02-20T16:44:41.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:41.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:41.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509446 2026-02-20T16:44:41.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:41.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509446 2026-02-20T16:44:41.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-20T16:44:41.638 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509446 2026-02-20T16:44:41.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:41.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509446 2026-02-20T16:44:41.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:41.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:41.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:42.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:42.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:42.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:42.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:42.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:42.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:42.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-20T16:44:42.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:44:42.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:44:42.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:44:42.558 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:44:42.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:44:42.764 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:44:42.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:44:42.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:44:42.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:44:42.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-20T16:44:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:44:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:44:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-20T16:44:42.806 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 9.5328e-05 s, 21.5 MB/s 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:42.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:43.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:43.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:43.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:43.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:43.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:44:43.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:43.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:44:43.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:44:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:44:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:44:43.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:43.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:43.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:43.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:43.439 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:43.440+0000 7f2c0ad55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:43.441 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:43.440+0000 7f2c0ad55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:43.442 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:43.440+0000 7f2c0ad55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:43.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:43.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:43.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:43.876 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:43.876+0000 7f2c0ad55a40 -1 Falling back to public interface 2026-02-20T16:44:44.526 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:44.524+0000 7f2c0ad55a40 -1 osd.0 41 log_to_monitors true 2026-02-20T16:44:44.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:44.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:44.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:44.743 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:44.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:44.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:44.905 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [5,41) [v2:127.0.0.1:6800/3101025327,v1:127.0.0.1:6801/3101025327] [v2:127.0.0.1:6802/3101025327,v1:127.0.0.1:6803/3101025327] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:44.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:44.907 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:44.907 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:44.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:44.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:44.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:45.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:45.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:45.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:45.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:45.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:45.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:45.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:45.179 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:45.179 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:45.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:45.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:45.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:45.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-20T16:44:45.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-20T16:44:45.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322' 2026-02-20T16:44:45.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:45.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:45.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757252 2026-02-20T16:44:45.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757252 2026-02-20T16:44:45.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322 1-163208757252' 2026-02-20T16:44:45.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:45.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:45.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-20T16:44:45.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-20T16:44:45.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322 1-163208757252 2-64424509449' 2026-02-20T16:44:45.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:45.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-193273528322 2026-02-20T16:44:45.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:45.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:45.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-193273528322 2026-02-20T16:44:45.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:45.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-20T16:44:45.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 193273528322' 2026-02-20T16:44:45.397 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 193273528322 2026-02-20T16:44:45.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:45.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 193273528322 2026-02-20T16:44:45.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:46.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:46.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:46.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-20T16:44:46.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:46.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-163208757252 2026-02-20T16:44:46.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:46.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:46.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-163208757252 2026-02-20T16:44:46.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:46.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757252 2026-02-20T16:44:46.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 163208757252' 2026-02-20T16:44:46.734 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 163208757252 2026-02-20T16:44:46.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:46.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757252 -lt 163208757252 2026-02-20T16:44:46.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:46.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-20T16:44:46.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:46.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:46.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-20T16:44:46.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:46.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-20T16:44:46.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-20T16:44:46.904 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509449 2026-02-20T16:44:46.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:47.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509449 -lt 64424509449 2026-02-20T16:44:47.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:47.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:47.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:47.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:47.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:47.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:47.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:47.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:47.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:47.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-20T16:44:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:44:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:44:47.816 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:44:47.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:44:48.021 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:44:48.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:44:48.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:44:48.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:44:48.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:44:48.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:48.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:48.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:48.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:48.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-20T16:44:48.558 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:48.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:48.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:48.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:48.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:44:48.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:44:48.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:48.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:48.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:48.685 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:48.683+0000 7f884b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:48.687 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:48.687+0000 7f884b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:48.688 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:48.687+0000 7f884b4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:48.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:44:48.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:48.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:48.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:49.072 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:49.071+0000 7f884b4b1a40 -1 Falling back to public interface 2026-02-20T16:44:49.652 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:49.651+0000 7f884b4b1a40 -1 osd.1 49 log_to_monitors true 2026-02-20T16:44:49.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:49.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:49.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:49.988 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:49.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:49.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:44:50.148 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 50 last_clean_interval [38,49) [v2:127.0.0.1:6810/789047762,v1:127.0.0.1:6811/789047762] [v2:127.0.0.1:6812/789047762,v1:127.0.0.1:6813/789047762] exists,up 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:44:50.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:50.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:50.150 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:50.150 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:50.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:50.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:50.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:50.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:50.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:50.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:50.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:50.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:50.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:50.424 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:50.424 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:50.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:50.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:50.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:50.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-20T16:44:50.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-20T16:44:50.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324' 2026-02-20T16:44:50.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:50.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:50.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266690 2026-02-20T16:44:50.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266690 2026-02-20T16:44:50.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324 1-227633266690' 2026-02-20T16:44:50.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:50.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:50.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509451 2026-02-20T16:44:50.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509451 2026-02-20T16:44:50.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324 1-227633266690 2-64424509451' 2026-02-20T16:44:50.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:50.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-193273528324 2026-02-20T16:44:50.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:50.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:50.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-193273528324 2026-02-20T16:44:50.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:50.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-20T16:44:50.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 193273528324' 2026-02-20T16:44:50.646 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 193273528324 2026-02-20T16:44:50.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:50.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528324 2026-02-20T16:44:50.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:51.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:51.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528324 2026-02-20T16:44:51.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:52.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:44:52.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:53.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528324 -lt 193273528324 2026-02-20T16:44:53.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:53.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266690 2026-02-20T16:44:53.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:53.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266690 2026-02-20T16:44:53.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:53.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266690 2026-02-20T16:44:53.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2026-02-20T16:44:53.143 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 227633266690 2026-02-20T16:44:53.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:53.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266690 -lt 227633266690 2026-02-20T16:44:53.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:53.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509451 2026-02-20T16:44:53.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:53.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:53.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509451 2026-02-20T16:44:53.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:53.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509451 2026-02-20T16:44:53.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2026-02-20T16:44:53.306 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509451 2026-02-20T16:44:53.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:53.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509451 -lt 64424509451 2026-02-20T16:44:53.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:53.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:53.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:53.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:53.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:53.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:53.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:53.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:53.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:53.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:53.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:53.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:53.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:53.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:53.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:44:54.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-20T16:44:54.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:44:54.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:44:54.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:44:54.257 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:44:54.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:44:54.461 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:44:54.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:44:54.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:44:54.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:44:54.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-20T16:44:54.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:44:54.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:44:54.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-20T16:44:54.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-20T16:44:54.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:54.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:54.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:54.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:54.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:44:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:55.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:55.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:55.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:55.166 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:55.163+0000 7f67303b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:55.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:55.167+0000 7f67303b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:55.168 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:55.167+0000 7f67303b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:55.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:44:55.310 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:44:55.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:55.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:55.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:44:55.540 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:55.539+0000 7f67303b8a40 -1 Falling back to public interface 2026-02-20T16:44:56.194 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:56.191+0000 7f67303b8a40 -1 osd.0 56 log_to_monitors true 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:44:56.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:44:56.629 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [45,56) [v2:127.0.0.1:6800/1429154844,v1:127.0.0.1:6801/1429154844] [v2:127.0.0.1:6802/1429154844,v1:127.0.0.1:6803/1429154844] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:44:56.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:44:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:44:56.631 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:44:56.631 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:44:56.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:44:56.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:44:56.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:44:56.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:44:56.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:44:56.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:44:56.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:44:56.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:44:56.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:56.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:44:56.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-20T16:44:56.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-20T16:44:56.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762' 2026-02-20T16:44:56.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:56.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:44:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266692 2026-02-20T16:44:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266692 2026-02-20T16:44:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762 1-227633266692' 2026-02-20T16:44:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:44:57.049 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:44:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509453 2026-02-20T16:44:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509453 2026-02-20T16:44:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762 1-227633266692 2-64424509453' 2026-02-20T16:44:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:57.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037762 2026-02-20T16:44:57.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:57.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:44:57.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037762 2026-02-20T16:44:57.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:57.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-20T16:44:57.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037762' 2026-02-20T16:44:57.127 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 257698037762 2026-02-20T16:44:57.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:57.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 257698037762 2026-02-20T16:44:57.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:44:58.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:44:58.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:44:58.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037762 -lt 257698037762 2026-02-20T16:44:58.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:58.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266692 2026-02-20T16:44:58.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:58.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:44:58.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266692 2026-02-20T16:44:58.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:58.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266692 2026-02-20T16:44:58.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266692' 2026-02-20T16:44:58.458 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 227633266692 2026-02-20T16:44:58.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:44:58.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266692 -lt 227633266692 2026-02-20T16:44:58.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:44:58.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509453 2026-02-20T16:44:58.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:44:58.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:44:58.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509453 2026-02-20T16:44:58.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:44:58.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509453 2026-02-20T16:44:58.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-20T16:44:58.622 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 64424509453 2026-02-20T16:44:58.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:44:58.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509453 -lt 64424509453 2026-02-20T16:44:58.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:44:58.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:58.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:58.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:44:58.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:44:58.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:44:59.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:44:59.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:44:59.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:44:59.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:44:59.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:44:59.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:44:59.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:44:59.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:44:59.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:44:59.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:44:59.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:44:59.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:44:59.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:44:59.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:44:59.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:44:59.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:44:59.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:59.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:44:59.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:44:59.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:44:59.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:44:59.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:44:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:44:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:44:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:44:59.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:44:59.982 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:59.979+0000 7f3b759fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:59.983 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:59.983+0000 7f3b759fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:44:59.984 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:44:59.983+0000 7f3b759fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:00.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:45:00.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:00.400 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:00.399+0000 7f3b759fea40 -1 Falling back to public interface 2026-02-20T16:45:01.087 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:01.087+0000 7f3b759fea40 -1 osd.2 62 log_to_monitors true 2026-02-20T16:45:01.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:01.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:01.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:01.279 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:01.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:01.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:45:01.442 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [15,62) [v2:127.0.0.1:6818/3141145734,v1:127.0.0.1:6819/3141145734] [v2:127.0.0.1:6820/3141145734,v1:127.0.0.1:6821/3141145734] exists,up 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:45:01.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:01.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:01.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:01.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:01.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:01.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:01.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:01.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:01.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:01.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:01.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:01.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:01.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:01.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:01.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:01.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037764 2026-02-20T16:45:01.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037764 2026-02-20T16:45:01.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764' 2026-02-20T16:45:01.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:01.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:01.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266694 2026-02-20T16:45:01.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266694 2026-02-20T16:45:01.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764 1-227633266694' 2026-02-20T16:45:01.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:01.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:01.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874242 2026-02-20T16:45:01.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874242 2026-02-20T16:45:01.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764 1-227633266694 2-279172874242' 2026-02-20T16:45:01.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:01.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037764 2026-02-20T16:45:01.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:01.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:01.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037764 2026-02-20T16:45:01.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:01.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037764 2026-02-20T16:45:01.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037764' 2026-02-20T16:45:01.942 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 257698037764 2026-02-20T16:45:01.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:02.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037762 -lt 257698037764 2026-02-20T16:45:02.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:03.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:03.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:03.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037764 -lt 257698037764 2026-02-20T16:45:03.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:03.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266694 2026-02-20T16:45:03.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:03.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:03.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266694 2026-02-20T16:45:03.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:03.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266694 2026-02-20T16:45:03.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2026-02-20T16:45:03.271 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 227633266694 2026-02-20T16:45:03.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:03.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266694 -lt 227633266694 2026-02-20T16:45:03.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:03.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-279172874242 2026-02-20T16:45:03.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:03.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:03.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-279172874242 2026-02-20T16:45:03.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874242 2026-02-20T16:45:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 279172874242' 2026-02-20T16:45:03.440 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 279172874242 2026-02-20T16:45:03.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:03.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874242 -lt 279172874242 2026-02-20T16:45:03.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:03.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:03.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:03.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:03.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:03.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:03.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:03.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:03.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:03.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:04.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:45:04.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:04.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:04.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:45:04.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:04.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:45:04.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:45:04.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:04.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:04.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:04.764 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:04.763+0000 7f1f60452a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:04.765 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:04.763+0000 7f1f60452a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:04.766 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:04.767+0000 7f1f60452a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:04.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:45:04.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:04.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:45:04.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:04.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:04.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:04.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:04.905 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:04.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:04.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:45:05.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:05.168 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:05.167+0000 7f1f60452a40 -1 Falling back to public interface 2026-02-20T16:45:05.831 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:05.831+0000 7f1f60452a40 -1 osd.2 66 log_to_monitors true 2026-02-20T16:45:06.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:06.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:06.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:06.067 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:06.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:06.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:45:06.228 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 70 up_thru 70 down_at 67 last_clean_interval [65,66) [v2:127.0.0.1:6818/859982065,v1:127.0.0.1:6819/859982065] [v2:127.0.0.1:6820/859982065,v1:127.0.0.1:6821/859982065] exists,up 46e7f9f6-53de-4fcd-8a87-91cdc0bd02b3 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:06.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:06.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:06.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:06.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:06.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:06.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:06.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:06.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:06.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:06.508 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:06.508 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:06.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:06.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:06.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:06.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037766 2026-02-20T16:45:06.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037766 2026-02-20T16:45:06.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766' 2026-02-20T16:45:06.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:06.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266696 2026-02-20T16:45:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266696 2026-02-20T16:45:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766 1-227633266696' 2026-02-20T16:45:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:06.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710722 2026-02-20T16:45:06.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710722 2026-02-20T16:45:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766 1-227633266696 2-300647710722' 2026-02-20T16:45:06.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:06.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037766 2026-02-20T16:45:06.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:06.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:06.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037766 2026-02-20T16:45:06.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:06.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037766 2026-02-20T16:45:06.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037766' 2026-02-20T16:45:06.734 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 257698037766 2026-02-20T16:45:06.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:06.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037765 -lt 257698037766 2026-02-20T16:45:06.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:07.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:07.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:08.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037765 -lt 257698037766 2026-02-20T16:45:08.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:09.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:09.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:09.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037766 -lt 257698037766 2026-02-20T16:45:09.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:09.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266696 2026-02-20T16:45:09.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:09.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:09.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266696 2026-02-20T16:45:09.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:09.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266696 2026-02-20T16:45:09.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266696' 2026-02-20T16:45:09.226 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 227633266696 2026-02-20T16:45:09.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:09.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266696 -lt 227633266696 2026-02-20T16:45:09.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:09.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710722 2026-02-20T16:45:09.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:09.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:09.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710722 2026-02-20T16:45:09.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:09.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710722 2026-02-20T16:45:09.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710722' 2026-02-20T16:45:09.396 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710722 2026-02-20T16:45:09.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:09.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710722 -lt 300647710722 2026-02-20T16:45:09.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:09.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:09.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:09.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:09.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:09.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:09.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:09.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:09.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:10.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:10.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:10.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:10.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-20T16:45:10.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:45:10.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-20T16:45:10.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-20T16:45:10.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:45:10.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:45:10.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-20T16:45:10.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:45:10.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:45:10.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:45:10.319 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:45:10.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:45:10.523 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:45:10.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:45:10.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:45:10.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:45:10.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-20T16:45:10.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:45:10.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:45:10.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records in 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records out 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.1335e-05 s, 66.8 MB/s 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:10.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:10.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:10.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:45:10.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:10.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:45:10.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:45:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:45:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:11.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:45:11.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:11.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:45:11.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:45:11.178 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:11.194 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:11.191+0000 7f1f39dd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:11.195 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:11.195+0000 7f1f39dd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:11.196 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:11.195+0000 7f1f39dd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:11.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:45:11.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:11.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:45:11.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:11.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:11.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:11.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:11.334 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:11.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:11.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:45:11.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:11.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:11.635+0000 7f1f39dd9a40 -1 Falling back to public interface 2026-02-20T16:45:12.190 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:12.187+0000 7f1f39dd9a40 -1 osd.1 73 log_to_monitors true 2026-02-20T16:45:12.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:12.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:12.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:12.497 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:12.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:12.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:45:12.662 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [53,73) [v2:127.0.0.1:6810/211818937,v1:127.0.0.1:6811/211818937] [v2:127.0.0.1:6812/211818937,v1:127.0.0.1:6813/211818937] exists,up 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:45:12.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:12.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:12.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:12.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:12.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:12.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:12.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:12.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:12.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:12.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:12.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:12.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:12.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:12.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:12.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:13.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037768 2026-02-20T16:45:13.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037768 2026-02-20T16:45:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768' 2026-02-20T16:45:13.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:13.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:13.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-20T16:45:13.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-20T16:45:13.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768 1-330712481794' 2026-02-20T16:45:13.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:13.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710724 2026-02-20T16:45:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710724 2026-02-20T16:45:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768 1-330712481794 2-300647710724' 2026-02-20T16:45:13.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:13.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037768 2026-02-20T16:45:13.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:13.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:13.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037768 2026-02-20T16:45:13.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:13.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037768 2026-02-20T16:45:13.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037768' 2026-02-20T16:45:13.158 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 257698037768 2026-02-20T16:45:13.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:13.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037767 -lt 257698037768 2026-02-20T16:45:13.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:14.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:14.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:14.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037768 -lt 257698037768 2026-02-20T16:45:14.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:14.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481794 2026-02-20T16:45:14.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:14.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:14.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481794 2026-02-20T16:45:14.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:14.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-20T16:45:14.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2026-02-20T16:45:14.480 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 330712481794 2026-02-20T16:45:14.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:14.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-20T16:45:14.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:14.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710724 2026-02-20T16:45:14.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:14.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710724 2026-02-20T16:45:14.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:14.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710724 2026-02-20T16:45:14.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710724' 2026-02-20T16:45:14.642 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710724 2026-02-20T16:45:14.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:14.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710724 -lt 300647710724 2026-02-20T16:45:14.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:14.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:14.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:14.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:14.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:14.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:15.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:15.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:45:15.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:45:15.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-20T16:45:15.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:45:15.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:45:15.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:45:15.548 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:45:15.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:45:15.753 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:45:15.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:45:15.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:45:15.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:45:15.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:15.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:15.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:16.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:16.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:45:16.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:16.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:16.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:45:16.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:45:16.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:16.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:16.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:16.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:16.451+0000 7f74bab68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:16.456 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:16.455+0000 7f74bab68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:16.457 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:16.455+0000 7f74bab68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:16.604 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:16.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:16.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:16.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:16.856 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:16.855+0000 7f74bab68a40 -1 Falling back to public interface 2026-02-20T16:45:17.483 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:17.483+0000 7f74bab68a40 -1 osd.0 80 log_to_monitors true 2026-02-20T16:45:17.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:17.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:17.765 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [60,80) [v2:127.0.0.1:6800/4266153611,v1:127.0.0.1:6801/4266153611] [v2:127.0.0.1:6802/4266153611,v1:127.0.0.1:6803/4266153611] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:17.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:17.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:17.923 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:17.923 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:17.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:17.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:17.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:18.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:18.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:18.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:18.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:18.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:18.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:18.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:18.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-20T16:45:18.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-20T16:45:18.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866' 2026-02-20T16:45:18.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:18.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:18.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481796 2026-02-20T16:45:18.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481796 2026-02-20T16:45:18.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796' 2026-02-20T16:45:18.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:18.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710726 2026-02-20T16:45:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710726 2026-02-20T16:45:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796 2-300647710726' 2026-02-20T16:45:18.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:18.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252866 2026-02-20T16:45:18.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:18.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:18.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252866 2026-02-20T16:45:18.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:18.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-20T16:45:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2026-02-20T16:45:18.423 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 360777252866 2026-02-20T16:45:18.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:18.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252866 -lt 360777252866 2026-02-20T16:45:18.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:18.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481796 2026-02-20T16:45:18.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:18.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:18.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481796 2026-02-20T16:45:18.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:18.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481796 2026-02-20T16:45:18.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481796' 2026-02-20T16:45:18.590 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 330712481796 2026-02-20T16:45:18.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:18.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481795 -lt 330712481796 2026-02-20T16:45:18.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:19.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:19.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:19.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481795 -lt 330712481796 2026-02-20T16:45:19.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:20.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:20.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:21.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481796 -lt 330712481796 2026-02-20T16:45:21.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:21.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710726 2026-02-20T16:45:21.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:21.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:21.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710726 2026-02-20T16:45:21.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:21.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710726 2026-02-20T16:45:21.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710726' 2026-02-20T16:45:21.089 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710726 2026-02-20T16:45:21.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:21.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710726 -lt 300647710726 2026-02-20T16:45:21.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:21.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:21.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:21.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:21.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:21.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:21.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:21.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:21.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:21.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:21.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:21.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:21.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:21.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:21.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:21.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:21.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:21.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:22.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:22.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:45:22.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:22.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:22.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:45:22.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:22.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:45:22.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:45:22.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:22.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:22.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:45:22.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:22.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:22.414 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:22.411+0000 7f7a14372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:22.416 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:22.415+0000 7f7a14372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:22.417 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:22.415+0000 7f7a14372a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:22.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:45:22.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:22.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:45:22.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:22.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:45:22.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:22.800 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:22.799+0000 7f7a14372a40 -1 Falling back to public interface 2026-02-20T16:45:23.457 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:23.455+0000 7f7a14372a40 -1 osd.1 85 log_to_monitors true 2026-02-20T16:45:23.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:23.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:23.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:23.707 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:23.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:23.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [77,85) [v2:127.0.0.1:6810/2912559593,v1:127.0.0.1:6811/2912559593] [v2:127.0.0.1:6812/2912559593,v1:127.0.0.1:6813/2912559593] exists,up 94229b16-0503-4483-afd2-4a432e39f2ed 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:23.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:23.874 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:23.874 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:23.875 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:23.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:23.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:23.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:23.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:23.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:23.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:23.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:23.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:23.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:24.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:24.149 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:24.149 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:24.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:24.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:24.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:24.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252868 2026-02-20T16:45:24.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252868 2026-02-20T16:45:24.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868' 2026-02-20T16:45:24.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:24.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-20T16:45:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-20T16:45:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346' 2026-02-20T16:45:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:24.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:24.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710728 2026-02-20T16:45:24.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710728 2026-02-20T16:45:24.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346 2-300647710728' 2026-02-20T16:45:24.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:24.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252868 2026-02-20T16:45:24.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:24.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:24.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252868 2026-02-20T16:45:24.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:24.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252868 2026-02-20T16:45:24.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252868' 2026-02-20T16:45:24.374 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 360777252868 2026-02-20T16:45:24.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:24.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252868 -lt 360777252868 2026-02-20T16:45:24.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:24.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089346 2026-02-20T16:45:24.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:24.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:24.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089346 2026-02-20T16:45:24.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:24.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-20T16:45:24.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2026-02-20T16:45:24.533 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 382252089346 2026-02-20T16:45:24.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:24.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089346 -lt 382252089346 2026-02-20T16:45:24.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:24.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710728 2026-02-20T16:45:24.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:24.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:24.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710728 2026-02-20T16:45:24.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:24.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710728 2026-02-20T16:45:24.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710728' 2026-02-20T16:45:24.693 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710728 2026-02-20T16:45:24.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710727 -lt 300647710728 2026-02-20T16:45:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:25.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:25.861 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:26.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710727 -lt 300647710728 2026-02-20T16:45:26.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:27.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:27.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:27.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710729 -lt 300647710728 2026-02-20T16:45:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:27.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:27.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:27.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:27.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:27.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:27.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:27.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:27.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:27.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:27.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:27.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:27.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-20T16:45:27.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:45:27.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:45:27.934 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:45:27.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:45:28.140 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:45:28.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:45:28.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:45:28.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:45:28.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-20T16:45:28.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:45:28.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:45:28.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-20T16:45:28.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-20T16:45:28.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:28.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:28.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:28.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:28.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-20T16:45:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:45:28.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:28.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:28.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:28.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:28.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:28.706 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:28.703+0000 7f4bf98afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:28.708 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:28.707+0000 7f4bf98afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:28.708 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:28.707+0000 7f4bf98afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:28.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:28.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:29.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:29.104 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:29.103+0000 7f4bf98afa40 -1 Falling back to public interface 2026-02-20T16:45:29.711 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:29.711+0000 7f4bf98afa40 -1 osd.0 92 log_to_monitors true 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:30.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:30.171 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6800/1863677355,v1:127.0.0.1:6801/1863677355] [v2:127.0.0.1:6802/1863677355,v1:127.0.0.1:6803/1863677355] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:45:30.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:30.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:30.173 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:30.173 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:30.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:30.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:30.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:30.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:30.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:30.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:30.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:30.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:30.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:30.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:30.451 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:30.451 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:30.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:30.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:30.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:30.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-20T16:45:30.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-20T16:45:30.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418' 2026-02-20T16:45:30.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:30.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:30.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089348 2026-02-20T16:45:30.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089348 2026-02-20T16:45:30.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089348' 2026-02-20T16:45:30.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:30.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:30.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710730 2026-02-20T16:45:30.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710730 2026-02-20T16:45:30.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089348 2-300647710730' 2026-02-20T16:45:30.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:30.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-412316860418 2026-02-20T16:45:30.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:30.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:30.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-412316860418 2026-02-20T16:45:30.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:30.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-20T16:45:30.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2026-02-20T16:45:30.670 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 412316860418 2026-02-20T16:45:30.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:30.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 412316860418 2026-02-20T16:45:30.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:31.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:31.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:32.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 412316860418 2026-02-20T16:45:32.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:33.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:33.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:33.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860418 -lt 412316860418 2026-02-20T16:45:33.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:33.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089348 2026-02-20T16:45:33.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:33.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:33.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089348 2026-02-20T16:45:33.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:33.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089348 2026-02-20T16:45:33.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089348' 2026-02-20T16:45:33.172 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 382252089348 2026-02-20T16:45:33.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:33.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089348 -lt 382252089348 2026-02-20T16:45:33.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:33.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710730 2026-02-20T16:45:33.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:33.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:33.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710730 2026-02-20T16:45:33.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:33.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710730 2026-02-20T16:45:33.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710730' 2026-02-20T16:45:33.342 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710730 2026-02-20T16:45:33.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:33.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710731 -lt 300647710730 2026-02-20T16:45:33.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:33.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:33.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:33.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:33.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:33.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:45:34.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:34.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:45:34.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:45:34.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:45:34.642 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:45:34.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:34.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:45:34.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:45:34.645 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:45:34.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:45:34.661 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:34.659+0000 7f13e5330a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:34.663 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:34.663+0000 7f13e5330a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:34.664 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:34.663+0000 7f13e5330a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:34.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:45:34.801 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:45:34.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:34.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:34.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:45:35.012 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:35.011+0000 7f13e5330a40 -1 Falling back to public interface 2026-02-20T16:45:35.671 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:45:35.671+0000 7f13e5330a40 -1 osd.0 97 log_to_monitors true 2026-02-20T16:45:35.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:45:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:45:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:45:35.963 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:45:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6800/411996925,v1:127.0.0.1:6801/411996925] [v2:127.0.0.1:6802/411996925,v1:127.0.0.1:6803/411996925] exists,up 6ba72c4e-583b-4176-8bf9-bc66306391c7 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:45:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:45:36.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:45:36.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:45:36.124 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:36.125 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:36.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:36.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:36.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:45:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:45:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:45:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:45:36.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:36.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:36.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:36.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-20T16:45:36.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-20T16:45:36.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898' 2026-02-20T16:45:36.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:36.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:36.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089350 2026-02-20T16:45:36.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089350 2026-02-20T16:45:36.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898 1-382252089350' 2026-02-20T16:45:36.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:36.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:36.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710733 2026-02-20T16:45:36.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710733 2026-02-20T16:45:36.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898 1-382252089350 2-300647710733' 2026-02-20T16:45:36.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:36.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696898 2026-02-20T16:45:36.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:36.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:36.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696898 2026-02-20T16:45:36.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:36.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-20T16:45:36.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2026-02-20T16:45:36.619 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 433791696898 2026-02-20T16:45:36.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:36.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 433791696898 2026-02-20T16:45:36.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:37.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:37.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:37.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 433791696898 2026-02-20T16:45:37.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:38.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:38.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:39.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696898 2026-02-20T16:45:39.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:39.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089350 2026-02-20T16:45:39.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:39.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:39.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089350 2026-02-20T16:45:39.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:39.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089350 2026-02-20T16:45:39.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2026-02-20T16:45:39.102 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 382252089350 2026-02-20T16:45:39.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:39.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089350 -lt 382252089350 2026-02-20T16:45:39.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:39.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710733 2026-02-20T16:45:39.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:39.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:39.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710733 2026-02-20T16:45:39.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:39.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710733 2026-02-20T16:45:39.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710733' 2026-02-20T16:45:39.281 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 300647710733 2026-02-20T16:45:39.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:39.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710733 -lt 300647710733 2026-02-20T16:45:39.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:45:39.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:39.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:39.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:45:39.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:45:39.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:45:39.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:45:39.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:45:39.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:45:40.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:45:40.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:45:40.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:45:40.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-20T16:45:40.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-20T16:45:40.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:45:40.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-20T16:45:40.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-20T16:45:40.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:45:40.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:45:40.164 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:40.165 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:40.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:40.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:40.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:40.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:45:40.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:45:40.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:45:40.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:45:40.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:40.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:40.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:40.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-20T16:45:40.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-20T16:45:40.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899' 2026-02-20T16:45:40.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:40.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:40.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089352 2026-02-20T16:45:40.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089352 2026-02-20T16:45:40.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899 1-382252089352' 2026-02-20T16:45:40.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:40.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:40.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710734 2026-02-20T16:45:40.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710734 2026-02-20T16:45:40.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899 1-382252089352 2-300647710734' 2026-02-20T16:45:40.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:40.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696899 2026-02-20T16:45:40.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:40.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:40.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696899 2026-02-20T16:45:40.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:40.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-20T16:45:40.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696899' 2026-02-20T16:45:40.787 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 433791696899 2026-02-20T16:45:40.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:40.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696899 2026-02-20T16:45:40.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:41.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:41.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:42.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696899 2026-02-20T16:45:42.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:43.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:45:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:43.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696900 -lt 433791696899 2026-02-20T16:45:43.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:43.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089352 2026-02-20T16:45:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:43.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:43.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089352 2026-02-20T16:45:43.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:43.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089352 2026-02-20T16:45:43.276 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 382252089352 2026-02-20T16:45:43.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089352' 2026-02-20T16:45:43.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:43.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089352 -lt 382252089352 2026-02-20T16:45:43.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:43.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710734 2026-02-20T16:45:43.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:43.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:43.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710734 2026-02-20T16:45:43.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:43.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710734 2026-02-20T16:45:43.436 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 300647710734 2026-02-20T16:45:43.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710734' 2026-02-20T16:45:43.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:43.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710735 -lt 300647710734 2026-02-20T16:45:43.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:45:43.603 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:45:43.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:45:43.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:45:43.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:45:43.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:45:43.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:45:43.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:45:43.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:43.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:43.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:43.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:43.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:43.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:45:43.981 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:45:43.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:43.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:43.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:43.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:43.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:43.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:44.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:44.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:45.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:45.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:45.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:45.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:45.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:45.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:45.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:45.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:45.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:46.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:46.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:46.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:46.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:46.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:46.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:46.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:46.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:46.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:47.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:47.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:47.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:47.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:47.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:47.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:47.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:47.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:47.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:48.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:48.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:48.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:48.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:48.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:48.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:48.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:48.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:49.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:49.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:49.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:49.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:49.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:49.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:49.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:49.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:49.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:45:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:45:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:45:51.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:45:44.469884+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:51.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:45:51.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-20T16:45:51.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:45:51.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-20T16:45:51.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:45:51.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-20T16:45:51.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-20T16:45:51.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:45:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=101 2026-02-20T16:45:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:45:51.181 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:45:51.182 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:45:51.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:45:51.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:45:51.199 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:45:51.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:45:51.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:45:51.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:45:51.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-20T16:45:51.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-20T16:45:51.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-20T16:45:51.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:45:51.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:45:51.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:45:51.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:45:51.227 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:45:51.227 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:45:51.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:45:51.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:45:51.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:45:51.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:45:51.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:45:51.681 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:45:51.682 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:45:51.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:45:51.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:51.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:45:51.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696903 2026-02-20T16:45:51.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696903 2026-02-20T16:45:51.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696903' 2026-02-20T16:45:51.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:51.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:45:51.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089355 2026-02-20T16:45:51.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089355 2026-02-20T16:45:51.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696903 1-382252089355' 2026-02-20T16:45:51.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:45:51.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:45:51.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710738 2026-02-20T16:45:51.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710738 2026-02-20T16:45:51.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696903 1-382252089355 2-300647710738' 2026-02-20T16:45:51.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:51.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696903 2026-02-20T16:45:51.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:51.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:45:51.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696903 2026-02-20T16:45:51.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:51.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696903 2026-02-20T16:45:51.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696903' 2026-02-20T16:45:51.904 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 433791696903 2026-02-20T16:45:51.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:52.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696901 -lt 433791696903 2026-02-20T16:45:52.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:45:53.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:45:53.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:45:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696903 -lt 433791696903 2026-02-20T16:45:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:53.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089355 2026-02-20T16:45:53.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:53.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:45:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089355 2026-02-20T16:45:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:53.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089355 2026-02-20T16:45:53.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089355' 2026-02-20T16:45:53.231 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 382252089355 2026-02-20T16:45:53.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:45:53.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089355 -lt 382252089355 2026-02-20T16:45:53.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:45:53.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-300647710738 2026-02-20T16:45:53.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:45:53.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:45:53.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-300647710738 2026-02-20T16:45:53.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:45:53.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710738 2026-02-20T16:45:53.393 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 300647710738 2026-02-20T16:45:53.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 300647710738' 2026-02-20T16:45:53.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:45:53.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710738 -lt 300647710738 2026-02-20T16:45:53.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:45:53.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:45:53.566 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:45:53.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:45:53.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:45:53.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:45:53.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:45:53.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:45:53.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:45:53.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:45:53.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:45:53.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:45:53.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:53.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:53.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:53.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:53.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:53.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:53.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:45:53.947 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-20T16:45:53.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:44:28.383338+0000 last_deep_scrub_stamp 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:53.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:55.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:55.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:55.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:55.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:55.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:55.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:55.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:55.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:55.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:56.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:56.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:56.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:56.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:56.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:56.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:56.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:56.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:56.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:57.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:57.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:57.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:57.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:57.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:57.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:57.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:57.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:57.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:58.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:58.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:58.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:58.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:58.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:45:59.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:45:59.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:45:59.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:45:59.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:45:59.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:45:59.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:45:59.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:45:59.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:44:28.383338+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:45:59.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:46:00.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:46:00.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:46:00.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:46:00.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:46:00.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:46:00.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:46:00.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:46:01.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:45:54.510843+0000 '>' 2026-02-20T16:44:28.383338+0000 2026-02-20T16:46:01.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:46:01.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-20T16:46:01.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:46:01.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-20T16:46:01.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:46:01.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-20T16:46:01.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-20T16:46:01.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-20T16:46:01.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:46:01.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=101 2026-02-20T16:46:01.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' false = true ']' 2026-02-20T16:46:01.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:46:01.152 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:46:01.152 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:46:01.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:46:01.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:46:01.171 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:46:01.172 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:46:01.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:46:01.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:46:01.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:46:01.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:46:01.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-20T16:46:01.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-20T16:46:01.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-20T16:46:01.400 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' removed 2026-02-20T16:46:01.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:46:01.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:46:01.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:46:01.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:01.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:01.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:01.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:01.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:01.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:01.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:46:01.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:46:01.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:46:01.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:46:01.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:46:01.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:46:01.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:46:01.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:46:01.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:46:01.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:46:01.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:46:01.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:46:01.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:46:01.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:46:01.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:46:01.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:46:01.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:01.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:01.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:01.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:01.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:01.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:46:01.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:46:01.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:46:01.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:46:01.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:46:01.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:46:01.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:46:01.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:46:01.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:46:01.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:46:01.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:46:01.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:46:01.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:46:01.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:46:01.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:46:01.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:46:01.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:46:01.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:46:01.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:46:01.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:46:01.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2026-02-20T16:46:01.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=true 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:46:01.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:46:01.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:46:01.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:46:01.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:46:01.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:46:01.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:46:01.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:46:01.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:46:01.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:46:01.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:46:01.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:46:01.895 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:46:01.896 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:46:01.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:46:01.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:46:02.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:46:02.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:02.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:02.045 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:02.046 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:02.046 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:02.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:02.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:46:02.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:46:02.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:46:02.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:02.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:46:02.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:46:02.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:46:02.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:02.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f7ed217d-253d-408a-872b-2cfcc5d562f3' 2026-02-20T16:46:02.078 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:02.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:46:02.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKj5hpbRQjBRAAk9H/qBh1izBxerSQOaAK6A== 2026-02-20T16:46:02.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKj5hpbRQjBRAAk9H/qBh1izBxerSQOaAK6A=="}' 2026-02-20T16:46:02.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f7ed217d-253d-408a-872b-2cfcc5d562f3 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:46:02.177 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:46:02.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:46:02.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKj5hpbRQjBRAAk9H/qBh1izBxerSQOaAK6A== --osd-uuid f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:02.203 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:02.202+0000 7f4ba9a0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:02.205 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:02.202+0000 7f4ba9a0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:02.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:02.202+0000 7f4ba9a0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:02.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:02.202+0000 7f4ba9a0ca40 -1 bdev(0x55a78d742c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:46:02.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:02.202+0000 7f4ba9a0ca40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:46:03.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:46:03.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:46:03.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:46:03.018 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:46:03.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:46:03.119 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:46:03.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:46:03.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:03.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:46:03.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:46:03.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:46:03.143 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:03.142+0000 7f8dd64f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:03.144 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:03.142+0000 7f8dd64f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:03.146 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:03.142+0000 7f8dd64f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:03.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:03.211 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:46:03.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:03.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:03.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:03.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:03.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:03.630+0000 7f8dd64f4a40 -1 Falling back to public interface 2026-02-20T16:46:04.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:04.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:04.303 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:46:04.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:04.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:04.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:04.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:04.552 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:04.550+0000 7f8dd64f4a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:46:05.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:05.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:05.392 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:46:05.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:46:05.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:05.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:05.544 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1829859313,v1:127.0.0.1:6801/1829859313] [v2:127.0.0.1:6802/1829859313,v1:127.0.0.1:6803/1829859313] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:05.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:46:05.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:46:05.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:46:05.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:05.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8386ec63-4833-487c-82a5-c8681ad7823b' 2026-02-20T16:46:05.549 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:05.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:46:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNj5hpZieLIRAAIJppLocmVBGBtQlzaaMvcw== 2026-02-20T16:46:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNj5hpZieLIRAAIJppLocmVBGBtQlzaaMvcw=="}' 2026-02-20T16:46:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8386ec63-4833-487c-82a5-c8681ad7823b -i td/osd-scrub-repair/1/new.json 2026-02-20T16:46:05.719 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:46:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:46:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNj5hpZieLIRAAIJppLocmVBGBtQlzaaMvcw== --osd-uuid 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:05.746 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:05.746+0000 7f1d446a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:05.748 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:05.746+0000 7f1d446a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:05.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:05.746+0000 7f1d446a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:05.750 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:05.746+0000 7f1d446a7a40 -1 bdev(0x561955bfcc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:46:05.750 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:05.746+0000 7f1d446a7a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:46:06.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:46:06.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:46:06.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:46:06.409 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:46:06.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:46:06.608 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:46:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:46:06.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:06.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:46:06.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:46:06.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:46:06.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:06.630+0000 7fb1ac4f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:06.634 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:06.634+0000 7fb1ac4f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:06.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:06.634+0000 7fb1ac4f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:06.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:46:06.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:06.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:46:06.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:06.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:06.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:06.771 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:46:06.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:06.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:06.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:06.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:07.056 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:07.054+0000 7fb1ac4f8a40 -1 Falling back to public interface 2026-02-20T16:46:07.628 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:07.626+0000 7fb1ac4f8a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:46:07.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:07.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:07.934 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:46:07.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:07.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:07.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:08.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:09.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:09.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:09.095 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:46:09.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:46:09.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:09.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:09.249 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2267080098,v1:127.0.0.1:6811/2267080098] [v2:127.0.0.1:6812/2267080098,v1:127.0.0.1:6813/2267080098] exists,up 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:46:09.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:09.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:09.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:46:09.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:46:09.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:09.257 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:09.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7380c611-97f8-4302-b4bd-21c52e0d45d9' 2026-02-20T16:46:09.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:46:09.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRj5hp/4MDEBAAWgkTdMx7NtZruLFYPPQpzQ== 2026-02-20T16:46:09.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRj5hp/4MDEBAAWgkTdMx7NtZruLFYPPQpzQ=="}' 2026-02-20T16:46:09.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7380c611-97f8-4302-b4bd-21c52e0d45d9 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:46:09.430 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:46:09.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:46:09.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRj5hp/4MDEBAAWgkTdMx7NtZruLFYPPQpzQ== --osd-uuid 7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:09.456 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:09.454+0000 7f200e40ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:09.458 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:09.454+0000 7f200e40ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:09.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:09.458+0000 7f200e40ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:09.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:09.458+0000 7f200e40ca40 -1 bdev(0x55a14b79ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:46:09.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:09.458+0000 7f200e40ca40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:46:10.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:46:10.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:46:10.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:46:10.114 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:46:10.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:46:10.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:46:10.309 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:46:10.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:10.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:46:10.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:46:10.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:46:10.331 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:10.330+0000 7f356c6dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:10.333 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:10.330+0000 7f356c6dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:10.334 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:10.330+0000 7f356c6dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:46:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:10.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:46:10.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:10.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:10.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:10.473 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:46:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:10.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:10.748 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:10.746+0000 7f356c6dba40 -1 Falling back to public interface 2026-02-20T16:46:11.388 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:11.386+0000 7f356c6dba40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:46:11.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:11.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:11.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:11.630 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:46:11.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:11.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:11.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:12.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:12.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:12.791 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:46:12.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:46:12.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:12.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:12.945 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2765214233,v1:127.0.0.1:6819/2765214233] [v2:127.0.0.1:6820/2765214233,v1:127.0.0.1:6821/2765214233] exists,up 7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:12.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:12.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:12.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:12.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-20T16:46:12.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:46:13.089 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:46:13.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:46:13.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:46:13.291 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:46:13.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:46:14.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:46:14.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-20T16:46:14.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-20T16:46:14.780 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T16:46:14.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:46:15.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2026-02-20T16:46:15.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:46:15.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:46:15.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T16:46:15.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:46:15.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-20T16:46:15.995 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:46:16.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:46:16.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:46:16.301 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:46:16.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:46:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T16:46:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T16:46:17.513 INFO:tasks.workunit.client.0.trial013.stderr:set pool 3 allow_ec_overwrites to true 2026-02-20T16:46:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:46:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:17.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:17.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:17.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:17.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:17.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:17.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:17.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:17.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:17.806 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:17.807 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:17.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:17.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:17.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:17.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T16:46:17.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T16:46:17.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T16:46:17.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:17.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:17.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:46:17.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:46:17.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-20T16:46:17.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:17.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:18.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-20T16:46:18.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-20T16:46:18.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147' 2026-02-20T16:46:18.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:18.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T16:46:18.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:18.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:18.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T16:46:18.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:18.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T16:46:18.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T16:46:18.027 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T16:46:18.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:18.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T16:46:18.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:19.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:19.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:19.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T16:46:19.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:19.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:46:19.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:19.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:19.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:46:19.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:19.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:46:19.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:46:19.351 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:46:19.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:19.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:46:19.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:19.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-20T16:46:19.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:19.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:19.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-20T16:46:19.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:19.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-20T16:46:19.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-20T16:46:19.511 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542147 2026-02-20T16:46:19.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:19.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-20T16:46:19.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:19.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:19.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:19.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:19.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:19.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:20.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:20.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:20.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:20.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:20.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:20.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:20.222 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:20.223 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:20.223 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:20.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:20.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:20.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:20.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:20.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:20.486 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:20.486 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:20.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:20.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:20.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:46:20.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:46:20.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:46:20.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:20.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:20.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:46:20.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:46:20.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-20T16:46:20.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:20.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:20.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T16:46:20.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T16:46:20.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-60129542148' 2026-02-20T16:46:20.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:20.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:46:20.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:20.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:20.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:46:20.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:20.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:46:20.709 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:46:20.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:46:20.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:20.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:46:20.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:21.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:21.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:22.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:46:22.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:23.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:46:23.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:23.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-20T16:46:23.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:23.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:46:23.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:23.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:23.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:46:23.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:23.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:46:23.204 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:46:23.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:46:23.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:23.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:46:23.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:23.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T16:46:23.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:23.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:23.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T16:46:23.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:23.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T16:46:23.376 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T16:46:23.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T16:46:23.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:23.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-20T16:46:23.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:23.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:23.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:23.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:23.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:23.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:23.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:23.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:23.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:23.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:23.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:23.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:23.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:23.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:23.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:24.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:24.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:24.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:24.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:46:24.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:46:24.303 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:46:24.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:46:24.509 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:46:24.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:46:24.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:46:24.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:46:24.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-20T16:46:24.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:46:24.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:46:24.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:24.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:24.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:24.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:46:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:24.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:25.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:25.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:46:25.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:25.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:46:25.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:46:25.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:25.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:25.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:25.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:25.166+0000 7f139f3a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:25.169 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:25.166+0000 7f139f3a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:25.170 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:25.170+0000 7f139f3a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:25.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:25.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:25.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:25.592 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:25.590+0000 7f139f3a0a40 -1 Falling back to public interface 2026-02-20T16:46:26.237 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:26.234+0000 7f139f3a0a40 -1 osd.1 32 log_to_monitors true 2026-02-20T16:46:26.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:26.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:26.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:26.471 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:26.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:26.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 33 last_clean_interval [9,32) [v2:127.0.0.1:6810/165822798,v1:127.0.0.1:6811/165822798] [v2:127.0.0.1:6812/165822798,v1:127.0.0.1:6813/165822798] exists,up 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:26.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:26.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:26.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:26.633 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:26.633 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:26.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:26.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:26.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:26.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:26.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:26.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:26.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:26.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:26.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:26.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:46:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:46:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:46:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:26.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:27.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822658 2026-02-20T16:46:27.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822658 2026-02-20T16:46:27.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-154618822658' 2026-02-20T16:46:27.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:27.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-154618822658 2-60129542150' 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:46:27.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:27.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:27.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:46:27.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:27.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:46:27.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:46:27.134 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836488 2026-02-20T16:46:27.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:27.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:46:27.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:28.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:28.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:28.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:46:28.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:28.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-154618822658 2026-02-20T16:46:28.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:28.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:28.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-154618822658 2026-02-20T16:46:28.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:28.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822658 2026-02-20T16:46:28.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2026-02-20T16:46:28.462 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 154618822658 2026-02-20T16:46:28.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:28.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822658 -lt 154618822658 2026-02-20T16:46:28.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:28.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-20T16:46:28.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:28.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:28.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-20T16:46:28.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:28.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-20T16:46:28.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-20T16:46:28.632 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542150 2026-02-20T16:46:28.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:28.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-20T16:46:28.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:29.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:29.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:29.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:29.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:29.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:29.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:29.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:29.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:29.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:29.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:29.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:29.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:46:29.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:46:29.564 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:46:29.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:46:29.769 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:46:29.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:46:29.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:46:29.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:46:29.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-20T16:46:29.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:46:29.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:46:29.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.4908e-05 s, 24.1 MB/s 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:29.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:29.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:46:30.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:30.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:30.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:30.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:46:30.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:46:30.438 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:46:30.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:30.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:46:30.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:46:30.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:30.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:30.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:30.461 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:30.458+0000 7f3aecca0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:30.463 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:30.462+0000 7f3aecca0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:30.464 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:30.462+0000 7f3aecca0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:30.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:30.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:30.856 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:30.854+0000 7f3aecca0a40 -1 Falling back to public interface 2026-02-20T16:46:31.417 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:31.414+0000 7f3aecca0a40 -1 osd.0 39 log_to_monitors true 2026-02-20T16:46:31.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:31.756 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:31.915 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [5,39) [v2:127.0.0.1:6800/3153828577,v1:127.0.0.1:6801/3153828577] [v2:127.0.0.1:6802/3153828577,v1:127.0.0.1:6803/3153828577] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:31.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:31.916 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:31.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:31.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:31.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:32.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:32.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:32.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:32.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:32.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:32.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:32.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:32.189 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:32.189 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:32.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:32.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:32.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:32.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-20T16:46:32.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-20T16:46:32.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730' 2026-02-20T16:46:32.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:32.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:32.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822660 2026-02-20T16:46:32.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822660 2026-02-20T16:46:32.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730 1-154618822660' 2026-02-20T16:46:32.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:32.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542152 2026-02-20T16:46:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542152 2026-02-20T16:46:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730 1-154618822660 2-60129542152' 2026-02-20T16:46:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:32.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-184683593730 2026-02-20T16:46:32.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:32.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-184683593730 2026-02-20T16:46:32.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-20T16:46:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 184683593730' 2026-02-20T16:46:32.413 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 184683593730 2026-02-20T16:46:32.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:32.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-20T16:46:32.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-154618822660 2026-02-20T16:46:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:32.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:32.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-154618822660 2026-02-20T16:46:32.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:32.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822660 2026-02-20T16:46:32.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 154618822660' 2026-02-20T16:46:32.587 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 154618822660 2026-02-20T16:46:32.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:32.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822659 -lt 154618822660 2026-02-20T16:46:32.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:33.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:33.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:33.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822659 -lt 154618822660 2026-02-20T16:46:33.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:34.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:46:34.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:35.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822660 -lt 154618822660 2026-02-20T16:46:35.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:35.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542152 2026-02-20T16:46:35.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:35.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:35.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542152 2026-02-20T16:46:35.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:35.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542152 2026-02-20T16:46:35.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-20T16:46:35.074 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542152 2026-02-20T16:46:35.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:35.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542152 2026-02-20T16:46:35.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:35.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:35.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:35.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:35.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:35.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:35.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:35.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:35.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:35.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-20T16:46:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:46:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:46:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-20T16:46:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:46:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:46:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:46:36.000 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:46:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:46:36.205 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:46:36.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:46:36.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:46:36.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:46:36.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:36.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:36.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:46:36.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:36.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-20T16:46:36.734 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-20T16:46:36.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:46:36.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:36.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:36.815 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:36.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:36.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:46:36.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:46:36.818 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:46:36.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:36.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:46:36.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:46:36.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:36.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:36.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:36.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:36.838+0000 7fe7bff08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:36.841 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:36.838+0000 7fe7bff08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:36.842 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:36.842+0000 7fe7bff08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:36.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:36.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:37.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:37.260 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:37.258+0000 7fe7bff08a40 -1 Falling back to public interface 2026-02-20T16:46:37.838 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:37.834+0000 7fe7bff08a40 -1 osd.1 47 log_to_monitors true 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 48 last_clean_interval [36,47) [v2:127.0.0.1:6810/3622352450,v1:127.0.0.1:6811/3622352450] [v2:127.0.0.1:6812/3622352450,v1:127.0.0.1:6813/3622352450] exists,up 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:38.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:38.293 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:38.293 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:38.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:38.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:38.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:38.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:38.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:38.577 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:38.577 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:38.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:38.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:38.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:38.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-20T16:46:38.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-20T16:46:38.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732' 2026-02-20T16:46:38.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:38.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:38.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332098 2026-02-20T16:46:38.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332098 2026-02-20T16:46:38.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732 1-219043332098' 2026-02-20T16:46:38.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:38.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:38.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-20T16:46:38.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-20T16:46:38.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732 1-219043332098 2-60129542155' 2026-02-20T16:46:38.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:38.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-184683593732 2026-02-20T16:46:38.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:38.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:38.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-184683593732 2026-02-20T16:46:38.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:38.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593732 2026-02-20T16:46:38.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 184683593732' 2026-02-20T16:46:38.801 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 184683593732 2026-02-20T16:46:38.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:38.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593731 -lt 184683593732 2026-02-20T16:46:38.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:39.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:39.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:40.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593731 -lt 184683593732 2026-02-20T16:46:40.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:41.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:46:41.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:41.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-20T16:46:41.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332098 2026-02-20T16:46:41.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:41.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:41.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332098 2026-02-20T16:46:41.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:41.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332098 2026-02-20T16:46:41.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332098' 2026-02-20T16:46:41.288 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 219043332098 2026-02-20T16:46:41.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:41.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332098 -lt 219043332098 2026-02-20T16:46:41.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:41.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-20T16:46:41.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:41.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-20T16:46:41.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:41.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-20T16:46:41.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-20T16:46:41.453 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542155 2026-02-20T16:46:41.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:41.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-20T16:46:41.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:41.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:41.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:41.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:41.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:41.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:41.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:41.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:41.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:41.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:41.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:41.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:41.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:41.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:41.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:46:42.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:46:42.427 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:46:42.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:46:42.630 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:46:42.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:46:42.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:46:42.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:46:42.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-20T16:46:42.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:46:42.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:46:42.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-20T16:46:42.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:42.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:42.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:42.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:42.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:43.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:46:43.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-20T16:46:43.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:43.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:43.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:46:43.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:46:43.283 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:46:43.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:43.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:46:43.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:46:43.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:43.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:43.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:43.299 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:43.297+0000 7f7ca5d4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:43.300 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:43.297+0000 7f7ca5d4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:43.301 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:43.297+0000 7f7ca5d4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:43.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:43.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:43.724 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:43.721+0000 7f7ca5d4da40 -1 Falling back to public interface 2026-02-20T16:46:44.382 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:44.377+0000 7f7ca5d4da40 -1 osd.0 54 log_to_monitors true 2026-02-20T16:46:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:44.606 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:46:44.761 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [43,54) [v2:127.0.0.1:6800/2475401004,v1:127.0.0.1:6801/2475401004] [v2:127.0.0.1:6802/2475401004,v1:127.0.0.1:6803/2475401004] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:46:44.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:44.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:44.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:44.763 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:44.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:44.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:44.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:44.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:44.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:44.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:44.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:44.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:44.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:45.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:45.035 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:45.035 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:45.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:45.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:45.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:45.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103170 2026-02-20T16:46:45.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103170 2026-02-20T16:46:45.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170' 2026-02-20T16:46:45.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:45.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:45.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332100 2026-02-20T16:46:45.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332100 2026-02-20T16:46:45.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-219043332100' 2026-02-20T16:46:45.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:45.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:45.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-20T16:46:45.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-20T16:46:45.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-219043332100 2-60129542157' 2026-02-20T16:46:45.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:45.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103170 2026-02-20T16:46:45.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:45.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:45.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103170 2026-02-20T16:46:45.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:45.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103170 2026-02-20T16:46:45.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103170' 2026-02-20T16:46:45.258 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 249108103170 2026-02-20T16:46:45.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:45.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593733 -lt 249108103170 2026-02-20T16:46:45.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:46.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:46.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:46.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103170 2026-02-20T16:46:46.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:46.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332100 2026-02-20T16:46:46.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:46.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:46.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332100 2026-02-20T16:46:46.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:46.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332100 2026-02-20T16:46:46.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332100' 2026-02-20T16:46:46.587 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 219043332100 2026-02-20T16:46:46.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:46.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332100 -lt 219043332100 2026-02-20T16:46:46.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:46.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-20T16:46:46.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:46.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:46.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-20T16:46:46.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:46.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-20T16:46:46.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-20T16:46:46.752 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542157 2026-02-20T16:46:46.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:46.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-20T16:46:46.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:46.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:46.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:47.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:47.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:47.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:47.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:47.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:47.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:47.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:47.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:47.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:47.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:46:47.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:46:48.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:48.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:48.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:46:48.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:46:48.082 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:46:48.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:48.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:46:48.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:46:48.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:48.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:48.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:48.098 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:48.097+0000 7f28fd642a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:48.099 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:48.097+0000 7f28fd642a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:48.100 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:48.097+0000 7f28fd642a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:48.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:46:48.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:48.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:48.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:48.448 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:48.445+0000 7f28fd642a40 -1 Falling back to public interface 2026-02-20T16:46:49.043 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:49.041+0000 7f28fd642a40 -1 osd.2 60 log_to_monitors true 2026-02-20T16:46:49.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:49.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:49.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:49.407 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:49.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:49.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:49.565 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [14,60) [v2:127.0.0.1:6818/3242054834,v1:127.0.0.1:6819/3242054834] [v2:127.0.0.1:6820/3242054834,v1:127.0.0.1:6821/3242054834] exists,up 7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:49.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:49.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:49.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:49.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:49.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:49.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:49.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:49.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:49.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:49.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:49.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:49.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:49.831 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:49.831 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:49.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:49.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:49.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:49.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103172 2026-02-20T16:46:49.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103172 2026-02-20T16:46:49.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172' 2026-02-20T16:46:49.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:49.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:49.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332102 2026-02-20T16:46:49.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332102 2026-02-20T16:46:49.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-219043332102' 2026-02-20T16:46:49.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:49.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:50.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939650 2026-02-20T16:46:50.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939650 2026-02-20T16:46:50.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-219043332102 2-270582939650' 2026-02-20T16:46:50.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:50.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103172 2026-02-20T16:46:50.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:50.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:50.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103172 2026-02-20T16:46:50.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:50.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103172 2026-02-20T16:46:50.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103172' 2026-02-20T16:46:50.055 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 249108103172 2026-02-20T16:46:50.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:50.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103172 2026-02-20T16:46:50.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:51.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:51.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:51.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103172 -lt 249108103172 2026-02-20T16:46:51.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:51.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332102 2026-02-20T16:46:51.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:51.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:51.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332102 2026-02-20T16:46:51.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:51.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332102 2026-02-20T16:46:51.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332102' 2026-02-20T16:46:51.380 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 219043332102 2026-02-20T16:46:51.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:51.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332102 -lt 219043332102 2026-02-20T16:46:51.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:51.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-270582939650 2026-02-20T16:46:51.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:51.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:51.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-270582939650 2026-02-20T16:46:51.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:51.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=270582939650 2026-02-20T16:46:51.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 270582939650' 2026-02-20T16:46:51.554 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 270582939650 2026-02-20T16:46:51.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:51.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939650 -lt 270582939650 2026-02-20T16:46:51.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:51.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:51.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:51.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:51.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:51.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:52.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:52.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:52.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:52.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:52.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:52.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:52.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:52.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:52.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:52.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:52.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:46:52.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:52.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:52.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:52.846 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:52.846 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:52.846 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:46:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:52.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:52.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:52.862 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:52.861+0000 7f7bd9bdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:52.863 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:52.861+0000 7f7bd9bdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:52.865 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:52.861+0000 7f7bd9bdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:53.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:46:53.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:53.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:53.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:53.280 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:53.277+0000 7f7bd9bdfa40 -1 Falling back to public interface 2026-02-20T16:46:53.878 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:53.873+0000 7f7bd9bdfa40 -1 osd.2 65 log_to_monitors true 2026-02-20T16:46:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:46:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:46:54.169 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:54.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:54.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:46:54.328 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [63,65) [v2:127.0.0.1:6818/2127736222,v1:127.0.0.1:6819/2127736222] [v2:127.0.0.1:6820/2127736222,v1:127.0.0.1:6821/2127736222] exists,up 7380c611-97f8-4302-b4bd-21c52e0d45d9 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:46:54.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:46:54.330 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:46:54.330 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:46:54.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:46:54.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:46:54.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:46:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:46:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:46:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:46:54.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:46:54.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:46:54.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:54.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:46:54.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103174 2026-02-20T16:46:54.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103174 2026-02-20T16:46:54.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174' 2026-02-20T16:46:54.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:54.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:46:54.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332104 2026-02-20T16:46:54.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332104 2026-02-20T16:46:54.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-219043332104' 2026-02-20T16:46:54.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:46:54.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:46:54.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-20T16:46:54.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-20T16:46:54.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-219043332104 2-296352743426' 2026-02-20T16:46:54.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:54.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103174 2026-02-20T16:46:54.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:54.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:46:54.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103174 2026-02-20T16:46:54.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:54.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103174 2026-02-20T16:46:54.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103174' 2026-02-20T16:46:54.842 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 249108103174 2026-02-20T16:46:54.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:54.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103172 -lt 249108103174 2026-02-20T16:46:54.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:55.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:46:56.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:56.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103172 -lt 249108103174 2026-02-20T16:46:56.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:46:57.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:46:57.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:46:57.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103174 -lt 249108103174 2026-02-20T16:46:57.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:57.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332104 2026-02-20T16:46:57.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:57.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:46:57.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332104 2026-02-20T16:46:57.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:57.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332104 2026-02-20T16:46:57.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332104' 2026-02-20T16:46:57.323 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 219043332104 2026-02-20T16:46:57.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:46:57.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332104 -lt 219043332104 2026-02-20T16:46:57.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:46:57.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743426 2026-02-20T16:46:57.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:46:57.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:46:57.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743426 2026-02-20T16:46:57.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:46:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-20T16:46:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2026-02-20T16:46:57.492 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743426 2026-02-20T16:46:57.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:46:57.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743426 -lt 296352743426 2026-02-20T16:46:57.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:46:57.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:57.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:57.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:46:57.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:46:57.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:46:57.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:46:57.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:46:57.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:46:57.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:46:57.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:46:58.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:46:58.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:46:58.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:46:58.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:46:58.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:46:58.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:46:58.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:46:58.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-20T16:46:58.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:46:58.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:46:58.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:46:58.428 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:46:58.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:46:58.631 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:46:58.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:46:58.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:46:58.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:46:58.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-20T16:46:58.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:46:58.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:46:58.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-20T16:46:58.664 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records in 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records out 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.7919e-05 s, 70.7 MB/s 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:58.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:46:58.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:58.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:46:59.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:59.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:46:59.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:46:59.262 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:46:59.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:46:59.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:46:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:46:59.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:46:59.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:46:59.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:46:59.279 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:59.277+0000 7f8d50bf7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:59.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:59.277+0000 7f8d50bf7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:59.282 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:59.277+0000 7f8d50bf7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:46:59.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:46:59.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:46:59.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:46:59.736 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:46:59.733+0000 7f8d50bf7a40 -1 Falling back to public interface 2026-02-20T16:47:00.410 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:00.405+0000 7f8d50bf7a40 -1 osd.1 72 log_to_monitors true 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:00.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:00.738 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [51,72) [v2:127.0.0.1:6810/876037865,v1:127.0.0.1:6811/876037865] [v2:127.0.0.1:6812/876037865,v1:127.0.0.1:6813/876037865] exists,up 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:47:00.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:00.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:00.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:00.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:47:00.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:00.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:00.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:00.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:00.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:00.740 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:00.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:00.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:00.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:00.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:00.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:00.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:00.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:00.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:00.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:01.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:01.015 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:01.016 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:01.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:01.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:01.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:01.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103176 2026-02-20T16:47:01.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103176 2026-02-20T16:47:01.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176' 2026-02-20T16:47:01.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:01.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514498 2026-02-20T16:47:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514498 2026-02-20T16:47:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-326417514498' 2026-02-20T16:47:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:01.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:01.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743428 2026-02-20T16:47:01.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743428 2026-02-20T16:47:01.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-326417514498 2-296352743428' 2026-02-20T16:47:01.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:01.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103176 2026-02-20T16:47:01.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:01.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:01.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103176 2026-02-20T16:47:01.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:01.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103176 2026-02-20T16:47:01.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103176' 2026-02-20T16:47:01.237 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 249108103176 2026-02-20T16:47:01.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:01.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103175 -lt 249108103176 2026-02-20T16:47:01.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:02.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:02.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:02.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103176 -lt 249108103176 2026-02-20T16:47:02.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:02.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-326417514498 2026-02-20T16:47:02.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:02.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:02.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-326417514498 2026-02-20T16:47:02.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:02.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514498 2026-02-20T16:47:02.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2026-02-20T16:47:02.573 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 326417514498 2026-02-20T16:47:02.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:02.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514498 -lt 326417514498 2026-02-20T16:47:02.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:02.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743428 2026-02-20T16:47:02.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:02.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:02.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743428 2026-02-20T16:47:02.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:02.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743428 2026-02-20T16:47:02.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743428' 2026-02-20T16:47:02.750 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743428 2026-02-20T16:47:02.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:02.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743428 -lt 296352743428 2026-02-20T16:47:02.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:47:02.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:02.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:03.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:47:03.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:47:03.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:47:03.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:47:03.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:47:03.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:47:03.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:47:03.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:47:03.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:47:03.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:47:03.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:03.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:03.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:47:03.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:47:03.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:47:03.654 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:47:03.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:47:03.858 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:47:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:47:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:47:03.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:47:03.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-20T16:47:03.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-20T16:47:03.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:47:03.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:47:03.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:03.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:03.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:03.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:03.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:04.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:04.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:04.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-20T16:47:04.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:04.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:04.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:04.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:47:04.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:47:04.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:47:04.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:47:04.522 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:47:04.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:04.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:47:04.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:47:04.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:47:04.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:47:04.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:47:04.538 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:04.533+0000 7f7038033a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:04.539 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:04.537+0000 7f7038033a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:04.540 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:04.537+0000 7f7038033a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:04.677 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:47:04.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:04.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:04.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:04.956 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:04.953+0000 7f7038033a40 -1 Falling back to public interface 2026-02-20T16:47:05.623 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:05.621+0000 7f7038033a40 -1 osd.0 79 log_to_monitors true 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:05.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:05.993 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [58,79) [v2:127.0.0.1:6800/1461403063,v1:127.0.0.1:6801/1461403063] [v2:127.0.0.1:6802/1461403063,v1:127.0.0.1:6803/1461403063] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:47:05.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:05.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:05.995 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:05.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:05.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:05.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:06.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:06.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:06.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:06.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:06.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:06.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:06.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:06.269 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:06.269 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:06.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:06.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:06.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:06.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285570 2026-02-20T16:47:06.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285570 2026-02-20T16:47:06.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570' 2026-02-20T16:47:06.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:06.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514500 2026-02-20T16:47:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514500 2026-02-20T16:47:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570 1-326417514500' 2026-02-20T16:47:06.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:06.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:06.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-20T16:47:06.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-20T16:47:06.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570 1-326417514500 2-296352743430' 2026-02-20T16:47:06.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:06.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285570 2026-02-20T16:47:06.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:06.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:06.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285570 2026-02-20T16:47:06.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:06.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285570 2026-02-20T16:47:06.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285570' 2026-02-20T16:47:06.490 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 356482285570 2026-02-20T16:47:06.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:06.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 356482285570 2026-02-20T16:47:06.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:07.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:07.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:07.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 356482285570 2026-02-20T16:47:07.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:08.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:47:08.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:08.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285570 -lt 356482285570 2026-02-20T16:47:08.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:08.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-326417514500 2026-02-20T16:47:08.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:08.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:08.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-326417514500 2026-02-20T16:47:08.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:08.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514500 2026-02-20T16:47:08.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 326417514500' 2026-02-20T16:47:08.982 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 326417514500 2026-02-20T16:47:08.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:09.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514500 -lt 326417514500 2026-02-20T16:47:09.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:09.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-20T16:47:09.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:09.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:09.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743430 2026-02-20T16:47:09.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:09.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743430 2026-02-20T16:47:09.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-20T16:47:09.147 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743430 2026-02-20T16:47:09.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:09.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-20T16:47:09.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:47:09.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:09.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:47:09.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:47:09.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:47:09.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:47:09.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:47:09.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:47:09.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:09.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:47:09.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:09.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:10.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:10.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:47:10.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:47:10.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:47:10.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:47:10.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:47:10.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:47:10.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-20T16:47:10.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:47:10.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:10.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:47:10.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:47:10.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:10.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:10.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:10.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:10.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:10.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:10.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:10.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:47:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:47:10.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:47:10.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:47:10.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:47:10.461 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:10.457+0000 7f441d86aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:10.463 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:10.461+0000 7f441d86aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:10.464 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:10.461+0000 7f441d86aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:10.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:10.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:10.884 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:10.881+0000 7f441d86aa40 -1 Falling back to public interface 2026-02-20T16:47:11.479 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:11.477+0000 7f441d86aa40 -1 osd.1 84 log_to_monitors true 2026-02-20T16:47:11.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:11.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:11.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:11.772 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:11.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:11.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:11.934 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [76,84) [v2:127.0.0.1:6810/1503462717,v1:127.0.0.1:6811/1503462717] [v2:127.0.0.1:6812/1503462717,v1:127.0.0.1:6813/1503462717] exists,up 8386ec63-4833-487c-82a5-c8681ad7823b 2026-02-20T16:47:11.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:11.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:11.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:11.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:47:11.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:11.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:11.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:11.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:11.936 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:11.936 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:11.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:11.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:11.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:12.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:12.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:12.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:12.207 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:12.207 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:12.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:12.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:12.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:12.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285572 2026-02-20T16:47:12.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285572 2026-02-20T16:47:12.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285572' 2026-02-20T16:47:12.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:12.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:12.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-20T16:47:12.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-20T16:47:12.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285572 1-377957122050' 2026-02-20T16:47:12.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:12.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:12.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743432 2026-02-20T16:47:12.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743432 2026-02-20T16:47:12.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285572 1-377957122050 2-296352743432' 2026-02-20T16:47:12.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:12.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285572 2026-02-20T16:47:12.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:12.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:12.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285572 2026-02-20T16:47:12.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:12.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285572 2026-02-20T16:47:12.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285572' 2026-02-20T16:47:12.433 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 356482285572 2026-02-20T16:47:12.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:12.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285572 -lt 356482285572 2026-02-20T16:47:12.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:12.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122050 2026-02-20T16:47:12.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:12.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:12.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122050 2026-02-20T16:47:12.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:12.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-20T16:47:12.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2026-02-20T16:47:12.598 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 377957122050 2026-02-20T16:47:12.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:12.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 377957122050 2026-02-20T16:47:12.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:13.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:13.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 377957122050 2026-02-20T16:47:13.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:14.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:47:14.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:15.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122050 2026-02-20T16:47:15.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:15.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743432 2026-02-20T16:47:15.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:15.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:15.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743432 2026-02-20T16:47:15.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743432 2026-02-20T16:47:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743432' 2026-02-20T16:47:15.093 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743432 2026-02-20T16:47:15.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:15.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743433 -lt 296352743432 2026-02-20T16:47:15.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:47:15.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:15.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:15.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:47:15.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:47:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:47:15.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:47:15.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:15.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:47:15.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:47:15.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-20T16:47:15.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:47:15.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:47:15.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:47:16.001 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:47:16.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:47:16.205 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:47:16.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:47:16.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:47:16.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:47:16.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-20T16:47:16.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-20T16:47:16.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-20T16:47:16.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-20T16:47:16.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-20T16:47:16.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:47:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:16.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:16.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:16.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:16.765 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:16.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:16.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:47:16.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:47:16.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:47:16.783 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:16.781+0000 7fd3a0975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:16.785 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:16.781+0000 7fd3a0975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:16.786 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:16.781+0000 7fd3a0975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:16.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:16.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:16.925 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:47:16.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:16.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:17.168 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:17.165+0000 7fd3a0975a40 -1 Falling back to public interface 2026-02-20T16:47:17.849 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:17.845+0000 7fd3a0975a40 -1 osd.0 91 log_to_monitors true 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:18.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [83,91) [v2:127.0.0.1:6800/1680119768,v1:127.0.0.1:6801/1680119768] [v2:127.0.0.1:6802/1680119768,v1:127.0.0.1:6803/1680119768] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:47:18.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:18.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:18.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:18.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:18.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:18.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:18.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:18.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:18.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:18.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:18.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:18.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-20T16:47:18.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-20T16:47:18.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122' 2026-02-20T16:47:18.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:18.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:18.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122052 2026-02-20T16:47:18.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122052 2026-02-20T16:47:18.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122 1-377957122052' 2026-02-20T16:47:18.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:18.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:18.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-20T16:47:18.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-20T16:47:18.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122 1-377957122052 2-296352743434' 2026-02-20T16:47:18.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:18.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-408021893122 2026-02-20T16:47:18.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:18.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:18.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-408021893122 2026-02-20T16:47:18.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:18.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-20T16:47:18.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2026-02-20T16:47:18.745 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 408021893122 2026-02-20T16:47:18.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:18.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 408021893122 2026-02-20T16:47:18.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:19.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:19.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:20.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 408021893122 2026-02-20T16:47:20.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:21.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:47:21.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:21.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893122 -lt 408021893122 2026-02-20T16:47:21.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:21.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122052 2026-02-20T16:47:21.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:21.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:21.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122052 2026-02-20T16:47:21.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:21.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122052 2026-02-20T16:47:21.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122052' 2026-02-20T16:47:21.242 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 377957122052 2026-02-20T16:47:21.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:21.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122052 -lt 377957122052 2026-02-20T16:47:21.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:21.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743434 2026-02-20T16:47:21.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:21.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:21.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743434 2026-02-20T16:47:21.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:21.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743434 2026-02-20T16:47:21.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743434' 2026-02-20T16:47:21.407 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743434 2026-02-20T16:47:21.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:21.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743435 -lt 296352743434 2026-02-20T16:47:21.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:47:21.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:21.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:47:21.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:47:21.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:47:21.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:47:21.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:47:21.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:47:21.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:21.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:47:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:22.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:47:22.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:22.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:22.698 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:22.698 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:22.698 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:22.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:22.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:22.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:47:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:22.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:47:22.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:47:22.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:47:22.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:47:22.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:47:22.717 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:22.713+0000 7f1f92d58a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:22.718 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:22.717+0000 7f1f92d58a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:22.719 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:22.717+0000 7f1f92d58a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:23.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:23.171 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:23.169+0000 7f1f92d58a40 -1 Falling back to public interface 2026-02-20T16:47:23.767 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:23.765+0000 7f1f92d58a40 -1 osd.0 96 log_to_monitors true 2026-02-20T16:47:24.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:24.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:24.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:24.022 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:24.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:24.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:24.179 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 97 last_clean_interval [95,96) [v2:127.0.0.1:6800/2274009007,v1:127.0.0.1:6801/2274009007] [v2:127.0.0.1:6802/2274009007,v1:127.0.0.1:6803/2274009007] exists,up f7ed217d-253d-408a-872b-2cfcc5d562f3 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:24.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:24.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:24.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:24.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:24.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:24.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:24.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:24.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:24.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:24.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:24.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:24.458 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:24.458 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:24.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:24.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:24.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:24.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729602 2026-02-20T16:47:24.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729602 2026-02-20T16:47:24.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729602' 2026-02-20T16:47:24.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:24.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:24.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122054 2026-02-20T16:47:24.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122054 2026-02-20T16:47:24.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729602 1-377957122054' 2026-02-20T16:47:24.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:24.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743437 2026-02-20T16:47:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743437 2026-02-20T16:47:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729602 1-377957122054 2-296352743437' 2026-02-20T16:47:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-429496729602 2026-02-20T16:47:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:24.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:24.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-429496729602 2026-02-20T16:47:24.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:24.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729602 2026-02-20T16:47:24.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2026-02-20T16:47:24.685 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 429496729602 2026-02-20T16:47:24.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:24.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 429496729602 2026-02-20T16:47:24.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:25.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:25.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:26.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 429496729602 2026-02-20T16:47:26.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:27.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:47:27.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:27.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729602 -lt 429496729602 2026-02-20T16:47:27.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:27.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122054 2026-02-20T16:47:27.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:27.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:27.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122054 2026-02-20T16:47:27.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:27.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122054 2026-02-20T16:47:27.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122054' 2026-02-20T16:47:27.185 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 377957122054 2026-02-20T16:47:27.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:27.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122054 -lt 377957122054 2026-02-20T16:47:27.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:27.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743437 2026-02-20T16:47:27.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:27.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:27.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743437 2026-02-20T16:47:27.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:27.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743437 2026-02-20T16:47:27.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743437' 2026-02-20T16:47:27.359 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 296352743437 2026-02-20T16:47:27.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:27.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743437 -lt 296352743437 2026-02-20T16:47:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:47:27.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:27.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:47:27.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:47:27.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:47:27.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:47:27.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:47:27.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:47:27.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:47:27.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:47:28.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:47:28.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:47:28.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:47:28.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-20T16:47:28.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-20T16:47:28.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:47:28.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-20T16:47:28.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-20T16:47:28.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:47:28.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:47:28.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:28.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:28.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:28.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:28.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:28.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:28.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:28.667 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:28.667 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:28.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:28.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:28.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729603 2026-02-20T16:47:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729603 2026-02-20T16:47:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729603' 2026-02-20T16:47:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:28.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:28.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122056 2026-02-20T16:47:28.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122056 2026-02-20T16:47:28.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729603 1-377957122056' 2026-02-20T16:47:28.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:28.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:28.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743438 2026-02-20T16:47:28.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743438 2026-02-20T16:47:28.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729603 1-377957122056 2-296352743438' 2026-02-20T16:47:28.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:28.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-429496729603 2026-02-20T16:47:28.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:28.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:28.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-429496729603 2026-02-20T16:47:28.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:28.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729603 2026-02-20T16:47:28.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 429496729603' 2026-02-20T16:47:28.890 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 429496729603 2026-02-20T16:47:28.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:29.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729602 -lt 429496729603 2026-02-20T16:47:29.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:30.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:30.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:30.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729602 -lt 429496729603 2026-02-20T16:47:30.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:31.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:47:31.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:31.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729604 -lt 429496729603 2026-02-20T16:47:31.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:31.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122056 2026-02-20T16:47:31.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:31.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:31.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122056 2026-02-20T16:47:31.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:31.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122056 2026-02-20T16:47:31.381 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 377957122056 2026-02-20T16:47:31.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122056' 2026-02-20T16:47:31.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122056 -lt 377957122056 2026-02-20T16:47:31.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:31.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743438 2026-02-20T16:47:31.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:31.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:31.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743438 2026-02-20T16:47:31.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:31.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743438 2026-02-20T16:47:31.554 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 296352743438 2026-02-20T16:47:31.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743438' 2026-02-20T16:47:31.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:31.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743439 -lt 296352743438 2026-02-20T16:47:31.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:47:31.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:47:31.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:47:31.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:47:31.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:47:31.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:47:31.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:47:31.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:31.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:31.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:31.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:31.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:31.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:47:32.089 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:47:32.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:32.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:32.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:33.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:33.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:33.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:33.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:33.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:33.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:33.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:33.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:33.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:34.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:34.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:34.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:34.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:34.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:34.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:34.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:34.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:34.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:35.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:35.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:35.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:35.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:35.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:35.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:35.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:35.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:35.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:36.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:36.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:36.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:36.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:36.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:36.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:36.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:36.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:36.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:37.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:37.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:37.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:37.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:37.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:37.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:37.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:38.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:38.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:39.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:39.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:39.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:47:39.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:39.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:47:39.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:39.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:47:39.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:47:32.478150+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:39.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:47:39.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-20T16:47:39.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:47:39.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-20T16:47:39.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:47:39.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-20T16:47:39.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-20T16:47:39.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:47:39.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=100 2026-02-20T16:47:39.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:47:39.295 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:47:39.296 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:47:39.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:47:39.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:47:39.311 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:47:39.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:47:39.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:47:39.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:47:39.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:47:39.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-20T16:47:39.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-20T16:47:39.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-20T16:47:39.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:47:39.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:47:39.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:47:39.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:47:39.337 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:39.337 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:39.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:39.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:39.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:47:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:47:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:47:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:47:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:39.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:39.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:39.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729607 2026-02-20T16:47:39.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729607 2026-02-20T16:47:39.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729607' 2026-02-20T16:47:39.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:39.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:39.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122059 2026-02-20T16:47:39.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122059 2026-02-20T16:47:39.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729607 1-377957122059' 2026-02-20T16:47:39.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:39.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:47:40.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743442 2026-02-20T16:47:40.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743442 2026-02-20T16:47:40.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-429496729607 1-377957122059 2-296352743442' 2026-02-20T16:47:40.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:40.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-429496729607 2026-02-20T16:47:40.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:40.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:40.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-429496729607 2026-02-20T16:47:40.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:40.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729607 2026-02-20T16:47:40.012 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 429496729607 2026-02-20T16:47:40.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 429496729607' 2026-02-20T16:47:40.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:40.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729605 -lt 429496729607 2026-02-20T16:47:40.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:47:41.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:47:41.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:41.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729607 -lt 429496729607 2026-02-20T16:47:41.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:41.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122059 2026-02-20T16:47:41.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:41.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:47:41.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122059 2026-02-20T16:47:41.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:41.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122059 2026-02-20T16:47:41.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122059' 2026-02-20T16:47:41.338 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 377957122059 2026-02-20T16:47:41.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:47:41.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122059 -lt 377957122059 2026-02-20T16:47:41.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:41.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743442 2026-02-20T16:47:41.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:41.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:47:41.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743442 2026-02-20T16:47:41.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:41.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743442 2026-02-20T16:47:41.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743442' 2026-02-20T16:47:41.506 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 296352743442 2026-02-20T16:47:41.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:47:41.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743442 -lt 296352743442 2026-02-20T16:47:41.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:47:41.674 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:47:41.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:47:41.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:47:41.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:47:41.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:47:41.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:47:41.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:41.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:41.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:41.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:41.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:41.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:47:42.054 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:46:16.300921+0000 last_deep_scrub_stamp 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:42.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:42.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:42.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:42.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:42.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:43.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:43.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:43.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:43.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:43.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:43.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:43.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:44.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:44.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:44.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:44.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:44.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:44.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:44.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:44.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:44.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:45.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:45.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:45.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:45.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:45.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:45.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:45.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:45.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:45.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:46.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:46.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:46.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:46.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:46.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:46.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:46.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:46.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:46.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:47.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:47.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:47.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:47.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:47.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:47.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:47.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:48.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:46:16.300921+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:48.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:47:49.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:47:49.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:47:49.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:47:49.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:47:49.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:47:49.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:47:49.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:47:49.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:47:42.406752+0000 '>' 2026-02-20T16:46:16.300921+0000 2026-02-20T16:47:49.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:47:49.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-20T16:47:49.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:47:49.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-20T16:47:49.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:47:49.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-20T16:47:49.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-20T16:47:49.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-20T16:47:49.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=100 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' true = true ']' 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:47:49.243 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:47:49.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:47:49.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:47:49.261 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:47:49.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-20T16:47:49.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:47:49.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:47:49.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:47:49.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-20T16:47:49.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-20T16:47:49.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-20T16:47:49.495 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' removed 2026-02-20T16:47:49.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:47:49.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:47:49.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:47:49.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:49.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:49.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:49.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:49.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:49.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:49.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:47:49.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:47:49.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:47:49.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:47:49.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:47:49.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:47:49.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:47:49.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:47:49.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:47:49.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:47:49.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:47:49.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:47:49.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:47:49.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:47:49.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:47:49.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:47:49.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:47:49.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:47:49.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:47:49.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:47:49.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:47:49.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:47:49.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:47:49.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:47:49.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:47:49.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:47:49.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:47:49.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:47:49.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:47:49.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:47:49.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:47:49.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:47:49.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:47:49.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:47:49.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:47:49.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:47:49.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:47:49.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:47:49.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:47:49.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:47:49.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:47:49.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:47:49.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:47:49.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:47:49.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:47:49.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1051: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2026-02-20T16:47:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: local total_objs=19 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1054: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:47:49.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:49.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T16:47:49.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:47:49.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:47:49.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:47:49.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:47:49.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:47:49.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:47:49.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:47:49.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:47:49.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:47:49.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:47:49.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:47:49.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:47:49.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:47:49.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:47:49.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:47:49.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:47:49.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:47:49.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:47:49.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:47:49.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1055: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2026-02-20T16:47:49.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:47:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:47:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:47:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:47:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:47:49.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:47:49.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:49.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:47:49.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:47:49.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1056: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2026-02-20T16:47:49.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:49.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:49.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:49.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:47:49.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:47:49.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:47:49.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 424c99bd-55ea-47a7-b5e9-87625992f8bd' 2026-02-20T16:47:49.984 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:47:49.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:47:49.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1kJhpm1Q3OxAAoi8W6quDQM509pb48VqOIg== 2026-02-20T16:47:49.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1kJhpm1Q3OxAAoi8W6quDQM509pb48VqOIg=="}' 2026-02-20T16:47:49.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 424c99bd-55ea-47a7-b5e9-87625992f8bd -i td/osd-scrub-repair/0/new.json 2026-02-20T16:47:50.080 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:47:50.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:47:50.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1kJhpm1Q3OxAAoi8W6quDQM509pb48VqOIg== --osd-uuid 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:47:50.106 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:50.104+0000 7f1543cafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:50.108 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:50.104+0000 7f1543cafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:50.108 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:50.104+0000 7f1543cafa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:50.109 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:50.104+0000 7f1543cafa40 -1 bdev(0x559c37932c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:47:50.110 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:50.104+0000 7f1543cafa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:47:50.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:47:50.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:47:50.876 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:47:50.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:47:50.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:47:50.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:47:50.981 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:47:50.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:50.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:47:50.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:47:50.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:47:51.008 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:51.004+0000 7ff6bc6eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:51.010 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:51.008+0000 7ff6bc6eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:51.011 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:51.008+0000 7ff6bc6eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:51.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:47:51.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:51.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:47:51.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:51.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:51.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:51.076 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:47:51.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:51.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:51.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:51.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:51.567 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:51.564+0000 7ff6bc6eca40 -1 Falling back to public interface 2026-02-20T16:47:52.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:52.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:52.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:52.169 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:47:52.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:52.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:52.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:52.571 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:52.568+0000 7ff6bc6eca40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:47:53.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:53.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:53.262 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:47:53.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:47:53.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:53.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:47:53.417 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/730093441,v1:127.0.0.1:6801/730093441] [v2:127.0.0.1:6802/730093441,v1:127.0.0.1:6803/730093441] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:47:53.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:47:53.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:47:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:47:53.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:47:53.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:47:53.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:47:53.423 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:47:53.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d1dd27eb-1c02-4ef5-896c-f6aa115eec8d' 2026-02-20T16:47:53.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:47:53.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5kJhp7isBGhAAkYPfJ06C/A8RufQonyoOuA== 2026-02-20T16:47:53.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5kJhp7isBGhAAkYPfJ06C/A8RufQonyoOuA=="}' 2026-02-20T16:47:53.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d1dd27eb-1c02-4ef5-896c-f6aa115eec8d -i td/osd-scrub-repair/1/new.json 2026-02-20T16:47:53.598 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:47:53.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:47:53.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5kJhp7isBGhAAkYPfJ06C/A8RufQonyoOuA== --osd-uuid d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:47:53.624 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:53.620+0000 7fbc48aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:53.625 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:53.620+0000 7fbc48aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:53.626 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:53.624+0000 7fbc48aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:53.627 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:53.624+0000 7fbc48aaba40 -1 bdev(0x55869756ac00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:47:53.627 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:53.624+0000 7fbc48aaba40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:47:54.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:47:54.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:47:54.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:47:54.246 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:47:54.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:47:54.451 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:47:54.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:47:54.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:47:54.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:47:54.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:47:54.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:47:54.468 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:54.464+0000 7f847d16fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:54.469 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:54.464+0000 7f847d16fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:54.470 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:54.468+0000 7f847d16fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:47:54.607 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:47:54.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:54.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:54.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:54.848 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:54.844+0000 7f847d16fa40 -1 Falling back to public interface 2026-02-20T16:47:55.478 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:47:55.472+0000 7f847d16fa40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:47:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:47:55.774 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:47:55.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:55.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:55.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:47:56.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:47:56.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:47:56.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:47:56.937 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:47:56.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:47:56.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:47:57.094 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/416634619,v1:127.0.0.1:6811/416634619] [v2:127.0.0.1:6812/416634619,v1:127.0.0.1:6813/416634619] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:47:57.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:47:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:47:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:47:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1058: TEST_corrupt_scrub_replicated: create_rbd_pool 2026-02-20T16:47:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:47:57.244 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:47:57.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:47:57.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:47:57.460 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:47:57.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:47:58.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:47:58.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1059: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-20T16:47:58.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:47:58.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:47:58.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:47:58.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:47:58.751 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:47:58.751 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:47:58.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:47:58.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:47:58.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:47:58.862 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:47:59.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:47:59.016 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:47:59.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:47:59.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:59.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:47:59.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:47:59.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:47:59.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:47:59.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:47:59.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:47:59.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-20T16:47:59.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-20T16:47:59.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-20T16:47:59.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:47:59.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:47:59.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:47:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:47:59.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:47:59.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:47:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:47:59.166 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:47:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:47:59.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:47:59.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T16:47:59.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:00.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:00.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:48:00.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:00.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-20T16:48:00.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:00.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:00.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-20T16:48:00.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:00.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-20T16:48:00.494 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672962 2026-02-20T16:48:00.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-20T16:48:00.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:00.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-20T16:48:00.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:00.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:00.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:00.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:48:00.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:00.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:01.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:48:01.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:01.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:01.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:01.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:48:01.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:01.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:01.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1061: TEST_corrupt_scrub_replicated: create_pool foo 1 2026-02-20T16:48:01.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-20T16:48:01.428 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T16:48:01.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:48:02.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2026-02-20T16:48:02.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-20T16:48:02.648 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' created 2026-02-20T16:48:02.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:48:03.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1063: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-20T16:48:03.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:03.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:03.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:03.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:03.662 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:03.663 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:03.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:03.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:03.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:03.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:03.940 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:03.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:03.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:03.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:04.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:48:04.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:48:04.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:48:04.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:04.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:04.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-20T16:48:04.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-20T16:48:04.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-20T16:48:04.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:04.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:48:04.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:04.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:04.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:48:04.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:04.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:48:04.090 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:48:04.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:48:04.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:04.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-20T16:48:04.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:05.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:05.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:05.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:48:05.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:05.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-20T16:48:05.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:05.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:05.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-20T16:48:05.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:05.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-20T16:48:05.415 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672964 2026-02-20T16:48:05.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-20T16:48:05.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:05.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-20T16:48:05.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:05.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:05.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:05.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:05.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:05.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:05.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:05.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:05.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:06.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:06.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:06.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:06.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:06.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:06.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:06.334 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:06.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:06.534 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:06.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:06.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:06.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:06.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-20T16:48:06.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-20T16:48:06.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:06.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:06.821 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:06.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:07.026 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:07.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:07.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:07.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:07.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-20T16:48:07.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:07.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:07.315 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:07.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:07.520 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:07.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:07.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-20T16:48:07.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:07.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:07.818 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:07.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:08.023 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:08.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:08.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:08.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:08.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-20T16:48:08.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-20T16:48:08.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:08.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:08.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:08.325 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:08.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:08.528 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:08.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:08.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:08.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:08.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-20T16:48:08.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:08.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:08.811 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:08.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:09.015 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:09.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:09.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:09.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:09.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-20T16:48:09.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-20T16:48:09.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:09.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-20T16:48:09.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-20T16:48:09.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:09.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:09.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-20T16:48:09.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:09.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:09.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:09.299 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:09.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:09.505 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:09.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:09.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:09.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:09.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-20T16:48:09.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:09.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-20T16:48:09.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:09.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:09.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:09.795 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:09.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:10.001 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:10.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:10.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:10.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:10.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-20T16:48:10.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-20T16:48:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-20T16:48:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-20T16:48:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-20T16:48:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:10.304 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:10.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:10.507 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:10.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:10.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:10.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:10.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-20T16:48:10.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-20T16:48:10.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-20T16:48:10.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:10.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:10.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:10.795 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:10.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:11.001 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:11.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:11.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-20T16:48:11.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:11.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:11.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:11.288 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:11.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:11.494 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:11.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:11.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:11.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:11.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-20T16:48:11.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-20T16:48:11.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:11.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-20T16:48:11.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-20T16:48:11.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:11.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:11.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-20T16:48:11.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:11.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:11.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:11.786 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:11.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:11.992 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:12.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:12.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:12.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:12.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-20T16:48:12.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:12.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:12.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:12.271 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:12.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:12.476 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:12.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:12.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:12.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:12.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-20T16:48:12.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-20T16:48:12.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:12.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:12.757 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:12.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:12.961 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:12.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:12.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:12.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:12.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-20T16:48:13.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-20T16:48:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-20T16:48:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-20T16:48:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:13.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-20T16:48:13.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:13.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:13.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:13.250 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:13.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:13.454 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:13.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:13.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:13.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:13.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-20T16:48:13.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-20T16:48:13.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:13.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-20T16:48:13.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-20T16:48:13.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:13.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:13.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-20T16:48:13.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:13.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:13.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:13.751 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:13.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:13.955 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:13.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:13.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:13.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:13.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-20T16:48:14.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:14.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:14.240 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:14.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:14.445 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:14.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:14.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:14.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:14.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-20T16:48:14.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-20T16:48:14.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:14.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:14.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:14.746 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:14.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:14.950 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:14.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:14.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-20T16:48:15.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:48:15.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:48:15.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:48:15.246 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:48:15.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:48:15.451 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:48:15.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:48:15.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:48:15.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:48:15.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-20T16:48:15.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-20T16:48:15.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-20T16:48:15.546 INFO:tasks.workunit.client.0.trial013.stderr:1025+0 records in 2026-02-20T16:48:15.546 INFO:tasks.workunit.client.0.trial013.stderr:1025+0 records out 2026-02-20T16:48:15.546 INFO:tasks.workunit.client.0.trial013.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00408821 s, 257 MB/s 2026-02-20T16:48:15.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-20T16:48:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-20T16:48:15.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-20T16:48:15.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-20T16:48:15.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-20T16:48:15.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:48:15.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:48:15.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-20T16:48:15.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2026-02-20T16:48:15.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-20T16:48:15.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-20T16:48:15.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:48:15.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:48:15.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: local primary=1 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:15.905 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:15.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:48:15.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-20T16:48:15.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-20T16:48:15.961 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:48:15.962 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-20T16:48:15.962 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:48:15.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2026-02-20T16:48:15.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:48:15.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:48:15.968 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:48:15.969 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:15.969 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:15.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:48:15.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-20T16:48:15.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-20T16:48:16.020 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:48:16.020 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-20T16:48:16.020 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:48:16.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-20T16:48:16.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:48:16.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:48:16.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:48:16.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:48:16.028 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:16.028 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:16.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:16.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:16.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:16.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:16.471 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:16.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:16.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:48:16.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:48:16.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:48:16.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:16.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-20T16:48:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-20T16:48:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2026-02-20T16:48:16.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:16.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:48:16.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:16.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:16.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:48:16.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:16.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:48:16.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:48:16.622 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T16:48:16.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:48:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:17.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:17.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:17.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-20T16:48:17.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:18.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:48:18.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:19.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:48:19.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:19.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-20T16:48:19.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:19.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:19.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-20T16:48:19.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:19.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-20T16:48:19.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-20T16:48:19.103 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672967 2026-02-20T16:48:19.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:19.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-20T16:48:19.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:48:19.276 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:48:19.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:48:19.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:48:19.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:48:19.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:48:19.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:48:19.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:48:19.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:48:19.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:48:19.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:48:19.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:48:19.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:19.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:48:19.653 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-20T16:48:19.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:48:02.648918+0000 last_deep_scrub_stamp 2026-02-20T16:48:19.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:48:19.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:48:19.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:48:19.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:48:19.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:02.648918+0000 '>' 2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:19.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:48:20.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:48:20.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:48:20.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:48:20.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:48:20.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:48:20.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:48:20.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:48:20.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:02.648918+0000 '>' 2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:20.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:48:21.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:48:21.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:48:21.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:48:21.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:48:21.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:48:21.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:48:21.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:48:22.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:02.648918+0000 '>' 2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:22.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:48:23.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:48:23.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:48:23.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:48:23.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:48:23.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:48:23.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:48:23.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:48:23.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:02.648918+0000 2026-02-20T16:48:23.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:48:23.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-20T16:48:23.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:23.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-20T16:48:23.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 1 % 2 2026-02-20T16:48:23.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:48:23.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:23.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2026-02-20T16:48:23.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1098: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:23.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:23.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:23.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:23.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:23.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:48:23.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:48:23.929 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:48:23.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:23.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:48:23.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:48:23.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:23.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:23.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:23.949 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:23.944+0000 7f5b8b884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:23.950 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:23.948+0000 7f5b8b884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:23.951 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:23.948+0000 7f5b8b884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:24.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:24.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:24.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:24.375 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:24.372+0000 7f5b8b884a40 -1 Falling back to public interface 2026-02-20T16:48:24.997 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:24.992+0000 7f5b8b884a40 -1 osd.1 63 log_to_monitors true 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:25.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [10,63) [v2:127.0.0.1:6810/949796891,v1:127.0.0.1:6811/949796891] [v2:127.0.0.1:6812/949796891,v1:127.0.0.1:6813/949796891] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:25.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:25.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:25.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:25.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:25.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:25.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:25.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:25.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:25.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:25.673 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:25.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:25.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:25.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:25.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:48:25.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:48:25.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:48:25.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:25.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:25.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-20T16:48:25.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-20T16:48:25.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-283467841538' 2026-02-20T16:48:25.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:25.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:48:25.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:25.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:25.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:48:25.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:25.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:48:25.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:48:25.821 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836491 2026-02-20T16:48:25.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:25.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:48:25.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:26.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:26.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:27.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:48:27.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:27.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-283467841538 2026-02-20T16:48:27.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:27.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:27.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-283467841538 2026-02-20T16:48:27.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:27.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-20T16:48:27.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2026-02-20T16:48:27.159 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 283467841538 2026-02-20T16:48:27.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:27.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-20T16:48:27.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:27.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:27.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:27.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:27.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:27.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:27.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:27.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:27.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-20T16:48:27.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 2 % 2 2026-02-20T16:48:27.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:48:27.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:27.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1104: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2026-02-20T16:48:27.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1105: TEST_corrupt_scrub_replicated: echo UVWXYZ 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:27.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:27.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:27.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:27.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:28.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:28.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:28.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:28.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:28.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:48:28.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:28.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:28.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:28.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:28.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:28.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:28.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:48:28.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:48:28.473 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:48:28.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:28.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:48:28.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:48:28.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:28.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:28.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:28.486 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:28.480+0000 7f931a02ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:28.487 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:28.484+0000 7f931a02ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:28.488 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:28.484+0000 7f931a02ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:28.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:48:28.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:28.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:28.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:28.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:28.903 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:28.900+0000 7f931a02ea40 -1 Falling back to public interface 2026-02-20T16:48:29.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:29.456+0000 7f931a02ea40 -1 osd.0 69 log_to_monitors true 2026-02-20T16:48:29.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:29.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:29.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:29.793 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:29.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:29.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,69) [v2:127.0.0.1:6800/627240334,v1:127.0.0.1:6801/627240334] [v2:127.0.0.1:6802/627240334,v1:127.0.0.1:6803/627240334] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:29.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:29.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:29.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:29.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:29.953 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:29.953 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:29.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:29.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:29.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:30.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:30.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:30.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:30.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:30.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:30.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:30.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:30.229 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:30.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:30.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:30.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:30.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-20T16:48:30.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-20T16:48:30.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314' 2026-02-20T16:48:30.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:30.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:30.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841540 2026-02-20T16:48:30.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841540 2026-02-20T16:48:30.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314 1-283467841540' 2026-02-20T16:48:30.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:30.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645314 2026-02-20T16:48:30.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:30.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:30.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645314 2026-02-20T16:48:30.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:30.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-20T16:48:30.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2026-02-20T16:48:30.380 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 309237645314 2026-02-20T16:48:30.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:30.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 309237645314 2026-02-20T16:48:30.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:31.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:31.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:31.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 309237645314 2026-02-20T16:48:31.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:32.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:48:32.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:32.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645314 -lt 309237645314 2026-02-20T16:48:32.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:32.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-283467841540 2026-02-20T16:48:32.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:32.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:32.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-283467841540 2026-02-20T16:48:32.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:32.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841540 2026-02-20T16:48:32.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 283467841540' 2026-02-20T16:48:32.869 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 283467841540 2026-02-20T16:48:32.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:33.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841540 -lt 283467841540 2026-02-20T16:48:33.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:33.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:33.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:33.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:33.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:33.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:33.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:33.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-20T16:48:33.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 3 % 2 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1111: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:33.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:33.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:33.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:48:33.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:33.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:33.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-20T16:48:34.058 INFO:tasks.workunit.client.0.trial013.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-20T16:48:34.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:48:34.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:34.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:34.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:34.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:34.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:34.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:34.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:34.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:34.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:34.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:34.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:34.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:48:34.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:34.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:48:34.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:48:34.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:34.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:34.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:34.205 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:34.204+0000 7fcda8227a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:34.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:34.208+0000 7fcda8227a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:34.207 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:34.208+0000 7fcda8227a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:34.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:34.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:34.555 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:34.556+0000 7fcda8227a40 -1 Falling back to public interface 2026-02-20T16:48:35.185 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:35.184+0000 7fcda8227a40 -1 osd.1 73 log_to_monitors true 2026-02-20T16:48:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:35.508 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:35.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:35.665 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6810/1963199549,v1:127.0.0.1:6811/1963199549] [v2:127.0.0.1:6812/1963199549,v1:127.0.0.1:6813/1963199549] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:48:35.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:35.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:35.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:35.666 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:35.667 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:35.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:35.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:35.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:35.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:35.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:35.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:35.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:35.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:35.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:35.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:35.945 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:35.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:35.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:35.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645316 2026-02-20T16:48:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645316 2026-02-20T16:48:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645316' 2026-02-20T16:48:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:36.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:36.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-20T16:48:36.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-20T16:48:36.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645316 1-330712481794' 2026-02-20T16:48:36.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:36.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645316 2026-02-20T16:48:36.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:36.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:36.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645316 2026-02-20T16:48:36.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645316 2026-02-20T16:48:36.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645316' 2026-02-20T16:48:36.096 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 309237645316 2026-02-20T16:48:36.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:36.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645314 -lt 309237645316 2026-02-20T16:48:36.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:37.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:37.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:37.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645316 -lt 309237645316 2026-02-20T16:48:37.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:37.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481794 2026-02-20T16:48:37.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:37.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:37.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481794 2026-02-20T16:48:37.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-20T16:48:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2026-02-20T16:48:37.428 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 330712481794 2026-02-20T16:48:37.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:37.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-20T16:48:37.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:37.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:37.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:37.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:37.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:37.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:37.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:37.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:37.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:37.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:38.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:38.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:38.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:38.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:38.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-20T16:48:38.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 4 % 2 2026-02-20T16:48:38.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:48:38.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:38.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1116: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:38.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:38.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:38.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:38.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:38.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:48:38.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:48:38.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:38.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:38.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:38.709 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:38.709 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:38.709 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:38.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:38.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:48:38.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:48:38.711 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:48:38.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:38.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:48:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:48:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:38.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:38.734 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:38.735+0000 7f3987fc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:38.736 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:38.735+0000 7f3987fc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:38.737 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:38.735+0000 7f3987fc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:38.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:48:38.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:38.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:48:38.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:38.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:39.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:39.103 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:39.103+0000 7f3987fc4a40 -1 Falling back to public interface 2026-02-20T16:48:39.737 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:39.735+0000 7f3987fc4a40 -1 osd.0 78 log_to_monitors true 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:40.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:40.192 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6800/4015604824,v1:127.0.0.1:6801/4015604824] [v2:127.0.0.1:6802/4015604824,v1:127.0.0.1:6803/4015604824] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:48:40.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:40.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:40.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:40.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:40.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:40.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:40.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:40.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:40.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:40.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:40.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:40.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:40.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:40.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:40.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:40.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:40.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:40.453 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:40.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:40.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:40.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318274 2026-02-20T16:48:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318274 2026-02-20T16:48:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274' 2026-02-20T16:48:40.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:40.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:40.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481796 2026-02-20T16:48:40.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481796 2026-02-20T16:48:40.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274 1-330712481796' 2026-02-20T16:48:40.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:40.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318274 2026-02-20T16:48:40.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:40.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:40.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318274 2026-02-20T16:48:40.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:40.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318274 2026-02-20T16:48:40.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2026-02-20T16:48:40.602 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 352187318274 2026-02-20T16:48:40.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:40.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 352187318274 2026-02-20T16:48:40.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:41.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:41.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:41.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 352187318274 2026-02-20T16:48:41.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:42.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:48:42.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:43.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318274 -lt 352187318274 2026-02-20T16:48:43.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:43.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481796 2026-02-20T16:48:43.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:43.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:43.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481796 2026-02-20T16:48:43.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:43.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481796 2026-02-20T16:48:43.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481796' 2026-02-20T16:48:43.095 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 330712481796 2026-02-20T16:48:43.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:43.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481796 -lt 330712481796 2026-02-20T16:48:43.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:43.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:43.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:43.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:43.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:43.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:43.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:43.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:43.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:43.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:43.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:43.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:43.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:43.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:43.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:43.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:43.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:43.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:43.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:43.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-20T16:48:43.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 5 % 2 2026-02-20T16:48:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:48:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1121: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:48:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:48:43.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:43.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:44.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:44.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:44.401 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:44.402 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:44.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:44.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:44.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:44.421 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:44.419+0000 7f1a0362ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:44.422 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:44.423+0000 7f1a0362ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:44.423 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:44.423+0000 7f1a0362ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:44.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:44.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:44.823 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:44.823+0000 7f1a0362ea40 -1 Falling back to public interface 2026-02-20T16:48:45.455 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:45.455+0000 7f1a0362ea40 -1 osd.1 84 log_to_monitors true 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:45.877 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/3459256293,v1:127.0.0.1:6811/3459256293] [v2:127.0.0.1:6812/3459256293,v1:127.0.0.1:6813/3459256293] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:45.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:45.879 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:45.879 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:45.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:45.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:45.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:45.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:45.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:45.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:45.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:45.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:45.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:46.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:46.158 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:46.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:46.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:46.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:46.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318276 2026-02-20T16:48:46.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318276 2026-02-20T16:48:46.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276' 2026-02-20T16:48:46.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:46.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276 1-377957122050' 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318276 2026-02-20T16:48:46.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:46.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:46.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318276 2026-02-20T16:48:46.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:46.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318276 2026-02-20T16:48:46.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318276' 2026-02-20T16:48:46.309 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 352187318276 2026-02-20T16:48:46.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:46.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318275 -lt 352187318276 2026-02-20T16:48:46.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:47.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:47.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:47.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318275 -lt 352187318276 2026-02-20T16:48:47.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:48.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:48:48.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:48.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318276 -lt 352187318276 2026-02-20T16:48:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:48.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122050 2026-02-20T16:48:48.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:48.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:48.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122050 2026-02-20T16:48:48.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:48.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-20T16:48:48.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2026-02-20T16:48:48.797 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 377957122050 2026-02-20T16:48:48.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:48.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122050 2026-02-20T16:48:48.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:48.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:48.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:49.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:49.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:49.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:49.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:49.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:49.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:49.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:49.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:49.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:49.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:49.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:49.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:49.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:49.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:49.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:49.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:49.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-20T16:48:49.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 6 % 2 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: echo extra 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:48:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:49.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:49.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:50.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:50.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:50.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:50.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:50.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:50.140 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:50.139+0000 7f0639b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:50.142 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:50.143+0000 7f0639b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:50.143 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:50.143+0000 7f0639b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:50.283 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:50.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:50.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:50.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:50.595 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:50.595+0000 7f0639b68a40 -1 Falling back to public interface 2026-02-20T16:48:51.238 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:51.239+0000 7f0639b68a40 -1 osd.0 89 log_to_monitors true 2026-02-20T16:48:51.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:51.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:51.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:51.449 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:51.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:51.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6800/3170386807,v1:127.0.0.1:6801/3170386807] [v2:127.0.0.1:6802/3170386807,v1:127.0.0.1:6803/3170386807] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:51.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:51.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:51.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:51.607 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:51.607 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:51.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:51.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:51.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:51.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:51.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:51.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:51.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:51.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:51.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:51.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:51.892 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:51.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:51.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:51.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:51.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958530 2026-02-20T16:48:51.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958530 2026-02-20T16:48:51.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530' 2026-02-20T16:48:51.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:51.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:52.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122052 2026-02-20T16:48:52.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122052 2026-02-20T16:48:52.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530 1-377957122052' 2026-02-20T16:48:52.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:52.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958530 2026-02-20T16:48:52.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:52.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:52.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958530 2026-02-20T16:48:52.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:52.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958530 2026-02-20T16:48:52.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2026-02-20T16:48:52.041 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 399431958530 2026-02-20T16:48:52.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:52.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318276 -lt 399431958530 2026-02-20T16:48:52.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:53.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:53.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958530 2026-02-20T16:48:53.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:53.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122052 2026-02-20T16:48:53.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:53.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:53.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122052 2026-02-20T16:48:53.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:53.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122052 2026-02-20T16:48:53.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122052' 2026-02-20T16:48:53.377 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 377957122052 2026-02-20T16:48:53.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:53.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122052 -lt 377957122052 2026-02-20T16:48:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:53.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:53.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:53.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:53.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:53.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:53.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:54.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:54.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:54.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:54.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1128: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2026-02-20T16:48:54.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:54.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-20T16:48:54.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 7 % 2 2026-02-20T16:48:54.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:48:54.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:54.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo -n newheader 2026-02-20T16:48:54.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:54.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:48:54.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:54.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:48:54.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:48:54.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:48:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:48:54.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:48:54.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:48:54.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:48:54.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:48:54.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:48:54.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:48:54.712 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:54.711+0000 7f14f3270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:54.714 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:54.715+0000 7f14f3270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:54.715 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:54.715+0000 7f14f3270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:48:54.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:55.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:48:55.115 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:55.115+0000 7f14f3270a40 -1 Falling back to public interface 2026-02-20T16:48:55.724 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:48:55.723+0000 7f14f3270a40 -1 osd.1 94 log_to_monitors true 2026-02-20T16:48:56.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:48:56.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:48:56.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:48:56.016 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:48:56.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:48:56.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:48:56.174 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 98 up_thru 98 down_at 95 last_clean_interval [88,94) [v2:127.0.0.1:6810/475077518,v1:127.0.0.1:6811/475077518] [v2:127.0.0.1:6812/475077518,v1:127.0.0.1:6813/475077518] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:48:56.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:48:56.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:48:56.176 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:48:56.176 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:48:56.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:48:56.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:48:56.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:48:56.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:48:56.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:48:56.451 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:48:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:48:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:56.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:48:56.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958532 2026-02-20T16:48:56.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958532 2026-02-20T16:48:56.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958532' 2026-02-20T16:48:56.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:48:56.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:48:56.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795010 2026-02-20T16:48:56.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795010 2026-02-20T16:48:56.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958532 1-420906795010' 2026-02-20T16:48:56.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:56.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958532 2026-02-20T16:48:56.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:56.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:48:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958532 2026-02-20T16:48:56.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958532 2026-02-20T16:48:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958532' 2026-02-20T16:48:56.602 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 399431958532 2026-02-20T16:48:56.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:56.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958532 2026-02-20T16:48:56.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:57.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:48:57.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:57.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958532 2026-02-20T16:48:57.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:48:58.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:48:58.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:48:59.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958532 -lt 399431958532 2026-02-20T16:48:59.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:48:59.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795010 2026-02-20T16:48:59.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:48:59.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:48:59.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795010 2026-02-20T16:48:59.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:48:59.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795010 2026-02-20T16:48:59.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2026-02-20T16:48:59.098 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 420906795010 2026-02-20T16:48:59.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:48:59.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795010 -lt 420906795010 2026-02-20T16:48:59.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:48:59.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:59.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:59.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:48:59.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:48:59.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:48:59.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:48:59.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:48:59.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:48:59.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:48:59.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:48:59.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:48:59.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:48:59.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:48:59.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:48:59.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:48:59.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:48:59.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:48:59.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2026-02-20T16:48:59.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:48:59.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-20T16:48:59.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 8 % 2 2026-02-20T16:48:59.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:48:59.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:48:59.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1139: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-20T16:48:59.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1140: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1143: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1144: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:48:59.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:48:59.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:48:59.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:48:59.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:48:59.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:48:59.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:48:59.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:00.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:00.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:00.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:49:00.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:49:00.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:00.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:00.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:49:00.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:00.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:00.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:00.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:49:00.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:49:00.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:00.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:00.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:00.507 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:00.507+0000 7f9c01e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:00.508 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:00.507+0000 7f9c01e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:00.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:00.507+0000 7f9c01e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:00.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:00.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:00.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:00.895 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:00.895+0000 7f9c01e28a40 -1 Falling back to public interface 2026-02-20T16:49:01.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:01.459+0000 7f9c01e28a40 -1 osd.0 99 log_to_monitors true 2026-02-20T16:49:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:01.805 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:01.963 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [93,99) [v2:127.0.0.1:6800/630023421,v1:127.0.0.1:6801/630023421] [v2:127.0.0.1:6802/630023421,v1:127.0.0.1:6803/630023421] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:49:01.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:01.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:01.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:01.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:01.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:01.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:01.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:01.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:01.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:01.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:01.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:01.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:01.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:02.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:02.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:02.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:02.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:02.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:02.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:02.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:02.245 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:02.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:02.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:02.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:02.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631490 2026-02-20T16:49:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631490 2026-02-20T16:49:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490' 2026-02-20T16:49:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:02.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:02.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795012 2026-02-20T16:49:02.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795012 2026-02-20T16:49:02.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490 1-420906795012' 2026-02-20T16:49:02.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:02.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631490 2026-02-20T16:49:02.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:02.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:02.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631490 2026-02-20T16:49:02.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:02.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631490 2026-02-20T16:49:02.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631490' 2026-02-20T16:49:02.398 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 442381631490 2026-02-20T16:49:02.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:02.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 442381631490 2026-02-20T16:49:02.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:03.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:03.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:03.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 442381631490 2026-02-20T16:49:03.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:04.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:04.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:04.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631490 -lt 442381631490 2026-02-20T16:49:04.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:04.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795012 2026-02-20T16:49:04.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:04.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:04.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795012 2026-02-20T16:49:04.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:04.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795012 2026-02-20T16:49:04.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795012' 2026-02-20T16:49:04.890 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 420906795012 2026-02-20T16:49:04.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:05.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795012 -lt 420906795012 2026-02-20T16:49:05.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:05.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:05.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:05.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:05.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:05.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:05.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:05.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:05.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:05.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:05.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:05.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:05.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:05.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:05.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:05.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1145: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:49:05.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:05.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:06.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:06.182 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:06.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:06.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:06.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:06.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:06.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:06.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:06.201 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:06.199+0000 7fa071247a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:06.202 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:06.203+0000 7fa071247a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:06.203 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:06.203+0000 7fa071247a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:06.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:06.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:06.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:06.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:06.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:06.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:06.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:06.348 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:06.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:06.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:06.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:06.567 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:06.567+0000 7fa071247a40 -1 Falling back to public interface 2026-02-20T16:49:07.148 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:07.147+0000 7fa071247a40 -1 osd.1 104 log_to_monitors true 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:07.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [98,104) [v2:127.0.0.1:6810/425538759,v1:127.0.0.1:6811/425538759] [v2:127.0.0.1:6812/425538759,v1:127.0.0.1:6813/425538759] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:07.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:07.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:07.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:07.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:07.665 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:07.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:07.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:07.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:07.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:07.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:07.943 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:07.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:07.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:07.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:08.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631492 2026-02-20T16:49:08.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631492 2026-02-20T16:49:08.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631492' 2026-02-20T16:49:08.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:08.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:08.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=463856467970 2026-02-20T16:49:08.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 463856467970 2026-02-20T16:49:08.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631492 1-463856467970' 2026-02-20T16:49:08.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:08.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631492 2026-02-20T16:49:08.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:08.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:08.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631492 2026-02-20T16:49:08.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:08.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631492 2026-02-20T16:49:08.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631492' 2026-02-20T16:49:08.093 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 442381631492 2026-02-20T16:49:08.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:08.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631490 -lt 442381631492 2026-02-20T16:49:08.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:09.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:09.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:09.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631492 -lt 442381631492 2026-02-20T16:49:09.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:09.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-463856467970 2026-02-20T16:49:09.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:09.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-463856467970 2026-02-20T16:49:09.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:09.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=463856467970 2026-02-20T16:49:09.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2026-02-20T16:49:09.426 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 463856467970 2026-02-20T16:49:09.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:09.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 463856467970 -lt 463856467970 2026-02-20T16:49:09.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:09.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:09.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:09.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:09.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:09.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:09.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:09.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:09.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:09.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:09.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:09.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:09.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:10.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1147: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:10.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:10.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:10.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:10.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:49:10.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:10.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:10.741 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:10.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:10.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:10.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:10.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:10.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:10.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:10.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:10.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:10.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:10.758 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:10.759+0000 7fcbf03bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:10.760 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:10.759+0000 7fcbf03bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:10.761 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:10.759+0000 7fcbf03bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:10.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:10.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:10.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:10.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:10.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:10.905 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:11.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:11.203 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:11.203+0000 7fcbf03bea40 -1 Falling back to public interface 2026-02-20T16:49:11.891 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:11.891+0000 7fcbf03bea40 -1 osd.1 109 log_to_monitors true 2026-02-20T16:49:12.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:12.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:12.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:12.069 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:12.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:12.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:12.222 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 113 up_thru 113 down_at 110 last_clean_interval [108,109) [v2:127.0.0.1:6810/1232776074,v1:127.0.0.1:6811/1232776074] [v2:127.0.0.1:6812/1232776074,v1:127.0.0.1:6813/1232776074] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:12.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:12.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:12.224 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:12.224 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:12.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:12.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:12.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:12.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:12.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:12.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:12.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:12.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:12.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:12.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:12.504 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:12.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:12.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:12.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:12.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631494 2026-02-20T16:49:12.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631494 2026-02-20T16:49:12.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631494' 2026-02-20T16:49:12.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:12.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:12.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=485331304450 2026-02-20T16:49:12.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 485331304450 2026-02-20T16:49:12.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631494 1-485331304450' 2026-02-20T16:49:12.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:12.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631494 2026-02-20T16:49:12.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:12.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:12.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631494 2026-02-20T16:49:12.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:12.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631494 2026-02-20T16:49:12.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631494' 2026-02-20T16:49:12.655 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 442381631494 2026-02-20T16:49:12.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:12.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631493 -lt 442381631494 2026-02-20T16:49:12.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:13.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:13.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:13.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631493 -lt 442381631494 2026-02-20T16:49:13.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:14.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:14.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:15.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631494 -lt 442381631494 2026-02-20T16:49:15.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-485331304450 2026-02-20T16:49:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:15.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:15.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-485331304450 2026-02-20T16:49:15.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:15.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=485331304450 2026-02-20T16:49:15.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 485331304450' 2026-02-20T16:49:15.154 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 485331304450 2026-02-20T16:49:15.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:15.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 485331304450 -lt 485331304450 2026-02-20T16:49:15.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:15.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:15.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:15.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:15.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:15.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:15.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:15.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:15.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:15.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:15.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:15.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:15.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:15.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-20T16:49:15.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:15.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-20T16:49:15.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 9 % 2 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:15.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:15.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:16.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:16.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:16.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-20T16:49:16.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:16.334 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:16.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:16.336 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:16.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:16.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:16.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:16.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:16.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:16.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:16.354 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:16.355+0000 7f7505dcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:16.356 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:16.355+0000 7f7505dcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:16.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:16.355+0000 7f7505dcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:16.502 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:16.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:16.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:16.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:16.723 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:16.723+0000 7f7505dcca40 -1 Falling back to public interface 2026-02-20T16:49:17.306 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:17.307+0000 7f7505dcca40 -1 osd.1 115 log_to_monitors true 2026-02-20T16:49:17.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:17.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:17.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:17.664 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:17.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:17.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:17.821 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 119 up_thru 119 down_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6810/1274381217,v1:127.0.0.1:6811/1274381217] [v2:127.0.0.1:6812/1274381217,v1:127.0.0.1:6813/1274381217] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:17.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:17.822 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:17.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:17.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:17.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:17.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:17.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:17.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:17.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:17.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:17.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:18.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:18.096 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:18.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:18.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:18.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:18.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631496 2026-02-20T16:49:18.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631496 2026-02-20T16:49:18.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496' 2026-02-20T16:49:18.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:18.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:18.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=511101108226 2026-02-20T16:49:18.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 511101108226 2026-02-20T16:49:18.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496 1-511101108226' 2026-02-20T16:49:18.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:18.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631496 2026-02-20T16:49:18.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:18.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:18.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631496 2026-02-20T16:49:18.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:18.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631496 2026-02-20T16:49:18.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631496' 2026-02-20T16:49:18.245 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 442381631496 2026-02-20T16:49:18.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:18.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631496 -lt 442381631496 2026-02-20T16:49:18.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:18.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-511101108226 2026-02-20T16:49:18.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:18.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:18.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-511101108226 2026-02-20T16:49:18.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:18.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=511101108226 2026-02-20T16:49:18.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 511101108226' 2026-02-20T16:49:18.413 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 511101108226 2026-02-20T16:49:18.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:18.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 511101108226 2026-02-20T16:49:18.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:19.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:19.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:19.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 511101108226 2026-02-20T16:49:19.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:20.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:20.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:20.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 511101108226 -lt 511101108226 2026-02-20T16:49:20.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:20.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:20.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:21.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:21.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:21.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:21.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:21.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:21.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:21.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:21.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:21.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: echo -n D 2026-02-20T16:49:21.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1154: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-20T16:49:21.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:21.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:21.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:21.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:21.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:21.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:22.111 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:22.111 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:22.113 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:22.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:22.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:22.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:22.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:22.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:22.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:22.134 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:22.135+0000 7fd8b06daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:22.136 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:22.135+0000 7fd8b06daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:22.137 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:22.135+0000 7fd8b06daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:22.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:22.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:22.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:22.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:22.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:22.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:22.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:22.276 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:22.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:22.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:22.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:22.559 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:22.559+0000 7fd8b06daa40 -1 Falling back to public interface 2026-02-20T16:49:23.234 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:23.235+0000 7fd8b06daa40 -1 osd.1 120 log_to_monitors true 2026-02-20T16:49:23.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:23.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:23.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:23.442 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:23.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:23.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:23.602 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 124 up_thru 124 down_at 121 last_clean_interval [119,120) [v2:127.0.0.1:6810/314662121,v1:127.0.0.1:6811/314662121] [v2:127.0.0.1:6812/314662121,v1:127.0.0.1:6813/314662121] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:23.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:23.604 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:23.604 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:23.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:23.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:23.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:23.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:23.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:23.879 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:23.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:23.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:23.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:23.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631498 2026-02-20T16:49:23.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631498 2026-02-20T16:49:23.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631498' 2026-02-20T16:49:23.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:23.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:24.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=532575944706 2026-02-20T16:49:24.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 532575944706 2026-02-20T16:49:24.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631498 1-532575944706' 2026-02-20T16:49:24.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:24.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631498 2026-02-20T16:49:24.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:24.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:24.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631498 2026-02-20T16:49:24.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:24.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631498 2026-02-20T16:49:24.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631498' 2026-02-20T16:49:24.033 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 442381631498 2026-02-20T16:49:24.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:24.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631497 -lt 442381631498 2026-02-20T16:49:24.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:25.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:25.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:25.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631498 -lt 442381631498 2026-02-20T16:49:25.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:25.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-532575944706 2026-02-20T16:49:25.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:25.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:25.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-532575944706 2026-02-20T16:49:25.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:25.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=532575944706 2026-02-20T16:49:25.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 532575944706' 2026-02-20T16:49:25.356 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 532575944706 2026-02-20T16:49:25.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:25.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 532575944706 -lt 532575944706 2026-02-20T16:49:25.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:25.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:25.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:25.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:25.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:25.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:25.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:25.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:25.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:25.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:26.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:26.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:26.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:26.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1156: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 10 % 2 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:49:26.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:26.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:26.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-20T16:49:26.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 11 % 2 2026-02-20T16:49:26.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:49:26.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:26.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:26.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-20T16:49:26.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 12 % 2 2026-02-20T16:49:26.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:49:26.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:26.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:26.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-20T16:49:26.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 13 % 2 2026-02-20T16:49:26.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:49:26.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:26.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:26.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 14 % 2 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:49:26.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:26.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:26.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:26.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:26.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:26.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:26.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:26.679 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:26.679 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:26.679 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:26.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:26.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:49:26.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:49:26.681 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:49:26.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:26.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:49:26.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:49:26.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:26.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:26.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:26.699 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:26.699+0000 7f92d523da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:26.700 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:26.699+0000 7f92d523da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:26.701 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:26.703+0000 7f92d523da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:26.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:26.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:27.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:27.091 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:27.091+0000 7f92d523da40 -1 Falling back to public interface 2026-02-20T16:49:27.690 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:27.691+0000 7f92d523da40 -1 osd.0 125 log_to_monitors true 2026-02-20T16:49:28.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:28.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:28.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:28.005 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:28.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:28.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 129 up_thru 129 down_at 126 last_clean_interval [103,125) [v2:127.0.0.1:6800/4147704719,v1:127.0.0.1:6801/4147704719] [v2:127.0.0.1:6802/4147704719,v1:127.0.0.1:6803/4147704719] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:28.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:28.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:28.163 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:28.163 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:28.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:28.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:28.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:28.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:28.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:28.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:28.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:28.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:28.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:28.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:28.442 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:28.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:28.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:28.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:28.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=554050781186 2026-02-20T16:49:28.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 554050781186 2026-02-20T16:49:28.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781186' 2026-02-20T16:49:28.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:28.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:28.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=532575944708 2026-02-20T16:49:28.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 532575944708 2026-02-20T16:49:28.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781186 1-532575944708' 2026-02-20T16:49:28.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:28.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-554050781186 2026-02-20T16:49:28.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:28.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:28.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-554050781186 2026-02-20T16:49:28.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:28.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=554050781186 2026-02-20T16:49:28.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 554050781186' 2026-02-20T16:49:28.592 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 554050781186 2026-02-20T16:49:28.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:28.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 554050781186 2026-02-20T16:49:28.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:29.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:29.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:29.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 554050781186 2026-02-20T16:49:29.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:30.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:30.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:31.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781186 -lt 554050781186 2026-02-20T16:49:31.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:31.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-532575944708 2026-02-20T16:49:31.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:31.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:31.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-532575944708 2026-02-20T16:49:31.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:31.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=532575944708 2026-02-20T16:49:31.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 532575944708' 2026-02-20T16:49:31.095 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 532575944708 2026-02-20T16:49:31.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:31.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 532575944708 -lt 532575944708 2026-02-20T16:49:31.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:31.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:31.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:31.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:31.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:31.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:31.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:31.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:31.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:31.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:31.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-20T16:49:32.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:32.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:32.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:32.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:32.409 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:32.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:32.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:32.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:32.425 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:32.427+0000 7f27acc24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:32.427 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:32.427+0000 7f27acc24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:32.428 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:32.427+0000 7f27acc24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:32.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:32.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:32.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:32.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:32.847 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:32.847+0000 7f27acc24a40 -1 Falling back to public interface 2026-02-20T16:49:33.497 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:33.494+0000 7f27acc24a40 -1 osd.1 130 log_to_monitors true 2026-02-20T16:49:33.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:33.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:33.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:33.727 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:33.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:33.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:33.885 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [124,130) [v2:127.0.0.1:6810/3691707407,v1:127.0.0.1:6811/3691707407] [v2:127.0.0.1:6812/3691707407,v1:127.0.0.1:6813/3691707407] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:33.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:33.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:33.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:33.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:33.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:33.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:33.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:33.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:33.887 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:33.887 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:33.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:33.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:33.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:34.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:34.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:34.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:34.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:34.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:34.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:34.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:34.159 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:34.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:34.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:34.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:34.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=554050781188 2026-02-20T16:49:34.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 554050781188 2026-02-20T16:49:34.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781188' 2026-02-20T16:49:34.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:34.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:34.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617666 2026-02-20T16:49:34.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617666 2026-02-20T16:49:34.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781188 1-575525617666' 2026-02-20T16:49:34.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:34.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-554050781188 2026-02-20T16:49:34.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:34.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:34.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-554050781188 2026-02-20T16:49:34.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:34.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=554050781188 2026-02-20T16:49:34.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 554050781188' 2026-02-20T16:49:34.313 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 554050781188 2026-02-20T16:49:34.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:34.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781187 -lt 554050781188 2026-02-20T16:49:34.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:35.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:35.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:35.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781187 -lt 554050781188 2026-02-20T16:49:35.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:36.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:36.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:36.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781188 -lt 554050781188 2026-02-20T16:49:36.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:36.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-575525617666 2026-02-20T16:49:36.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:36.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:36.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-575525617666 2026-02-20T16:49:36.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:36.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617666 2026-02-20T16:49:36.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 575525617666' 2026-02-20T16:49:36.807 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 575525617666 2026-02-20T16:49:36.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:36.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617666 -lt 575525617666 2026-02-20T16:49:36.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:36.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:36.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:37.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:37.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:37.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:37.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:37.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:37.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:37.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:37.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:37.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:37.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:37.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2026-02-20T16:49:37.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:37.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-20T16:49:37.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 15 % 2 2026-02-20T16:49:37.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:37.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:37.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:37.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:37.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-20T16:49:38.024 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:49:38.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:38.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:38.141 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:38.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:38.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:38.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:38.164 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:38.162+0000 7f5845944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:38.166 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:38.166+0000 7f5845944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:38.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:38.166+0000 7f5845944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:38.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:38.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:38.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:38.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:38.591 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:38.590+0000 7f5845944a40 -1 Falling back to public interface 2026-02-20T16:49:39.223 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:39.222+0000 7f5845944a40 -1 osd.1 136 log_to_monitors true 2026-02-20T16:49:39.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:39.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:39.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:39.469 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:39.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:39.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:39.624 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 140 up_thru 140 down_at 137 last_clean_interval [134,136) [v2:127.0.0.1:6810/1425684806,v1:127.0.0.1:6811/1425684806] [v2:127.0.0.1:6812/1425684806,v1:127.0.0.1:6813/1425684806] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:39.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:39.626 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:39.626 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:39.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:39.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:39.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:39.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:39.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:39.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:39.895 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:39.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:39.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:39.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=554050781190 2026-02-20T16:49:39.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 554050781190 2026-02-20T16:49:39.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781190' 2026-02-20T16:49:39.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:39.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=601295421442 2026-02-20T16:49:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 601295421442 2026-02-20T16:49:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-554050781190 1-601295421442' 2026-02-20T16:49:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:40.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-554050781190 2026-02-20T16:49:40.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:40.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:40.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-554050781190 2026-02-20T16:49:40.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:40.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=554050781190 2026-02-20T16:49:40.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 554050781190' 2026-02-20T16:49:40.044 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 554050781190 2026-02-20T16:49:40.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:40.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781189 -lt 554050781190 2026-02-20T16:49:40.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:41.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:41.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:41.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 554050781190 -lt 554050781190 2026-02-20T16:49:41.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:41.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-601295421442 2026-02-20T16:49:41.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:41.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:41.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-601295421442 2026-02-20T16:49:41.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:41.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=601295421442 2026-02-20T16:49:41.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2026-02-20T16:49:41.370 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 601295421442 2026-02-20T16:49:41.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:41.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 601295421442 -lt 601295421442 2026-02-20T16:49:41.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:41.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:41.537 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:41.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:41.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:41.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:41.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:42.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:42.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:42.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:42.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-20T16:49:42.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 16 % 2 2026-02-20T16:49:42.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:49:42.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:42.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:49:42.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:42.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:42.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:42.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:42.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:42.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-20T16:49:42.574 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:42.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:42.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:42.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:49:42.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:49:42.689 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:49:42.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:42.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:49:42.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:49:42.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:42.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:42.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:42.708 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:42.706+0000 7f5377b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:42.710 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:42.710+0000 7f5377b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:42.710 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:42.710+0000 7f5377b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:42.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:49:42.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:42.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:43.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:43.103 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:43.102+0000 7f5377b9da40 -1 Falling back to public interface 2026-02-20T16:49:43.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:43.690+0000 7f5377b9da40 -1 osd.0 142 log_to_monitors true 2026-02-20T16:49:44.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:44.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:44.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:44.016 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:44.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:44.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:44.172 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 146 up_thru 146 down_at 143 last_clean_interval [129,142) [v2:127.0.0.1:6800/2290161591,v1:127.0.0.1:6801/2290161591] [v2:127.0.0.1:6802/2290161591,v1:127.0.0.1:6803/2290161591] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:44.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:44.175 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:44.175 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:44.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:44.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:44.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:44.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:44.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:44.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:44.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:44.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:44.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:44.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:44.460 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:44.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:44.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:44.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:44.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=627065225218 2026-02-20T16:49:44.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 627065225218 2026-02-20T16:49:44.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-627065225218' 2026-02-20T16:49:44.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:44.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=601295421444 2026-02-20T16:49:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 601295421444 2026-02-20T16:49:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-627065225218 1-601295421444' 2026-02-20T16:49:44.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:44.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-627065225218 2026-02-20T16:49:44.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:44.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:44.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-627065225218 2026-02-20T16:49:44.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:44.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=627065225218 2026-02-20T16:49:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 627065225218' 2026-02-20T16:49:44.611 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 627065225218 2026-02-20T16:49:44.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 627065225218 2026-02-20T16:49:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:45.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:45.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:45.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 627065225218 2026-02-20T16:49:45.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:46.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:46.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:47.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 627065225218 -lt 627065225218 2026-02-20T16:49:47.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:47.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-601295421444 2026-02-20T16:49:47.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:47.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:47.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-601295421444 2026-02-20T16:49:47.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:47.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=601295421444 2026-02-20T16:49:47.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 601295421444' 2026-02-20T16:49:47.100 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 601295421444 2026-02-20T16:49:47.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:47.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 601295421444 -lt 601295421444 2026-02-20T16:49:47.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:47.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:47.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:47.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:47.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:47.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:47.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:47.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:47.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:47.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:47.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:47.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:47.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:48.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:48.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:49:48.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:48.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:48.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:48.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:48.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:48.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:49:48.313 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:49:48.314 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:48.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:48.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:48.432 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:48.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:48.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:48.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:48.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:48.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:48.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:48.452 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:48.450+0000 7f1dc8756a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:48.454 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:48.454+0000 7f1dc8756a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:48.455 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:48.454+0000 7f1dc8756a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:48.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:48.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:48.839 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:48.838+0000 7f1dc8756a40 -1 Falling back to public interface 2026-02-20T16:49:49.472 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:49.470+0000 7f1dc8756a40 -1 osd.1 147 log_to_monitors true 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:49.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:49.918 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 151 up_thru 151 down_at 148 last_clean_interval [140,147) [v2:127.0.0.1:6810/2069120312,v1:127.0.0.1:6811/2069120312] [v2:127.0.0.1:6812/2069120312,v1:127.0.0.1:6813/2069120312] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:49.919 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:49.920 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:49.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:49.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:49.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:50.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:50.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:50.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:50.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:50.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:50.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:50.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:50.193 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:50.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:50.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:50.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:50.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=627065225220 2026-02-20T16:49:50.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 627065225220 2026-02-20T16:49:50.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-627065225220' 2026-02-20T16:49:50.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:50.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:50.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=648540061698 2026-02-20T16:49:50.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 648540061698 2026-02-20T16:49:50.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-627065225220 1-648540061698' 2026-02-20T16:49:50.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:50.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-627065225220 2026-02-20T16:49:50.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:50.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:50.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-627065225220 2026-02-20T16:49:50.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:50.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=627065225220 2026-02-20T16:49:50.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 627065225220' 2026-02-20T16:49:50.346 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 627065225220 2026-02-20T16:49:50.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:50.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 627065225219 -lt 627065225220 2026-02-20T16:49:50.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:51.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:51.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:51.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 627065225219 -lt 627065225220 2026-02-20T16:49:51.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:52.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:49:52.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:52.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 627065225220 -lt 627065225220 2026-02-20T16:49:52.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:52.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-648540061698 2026-02-20T16:49:52.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:52.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:52.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-648540061698 2026-02-20T16:49:52.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:52.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=648540061698 2026-02-20T16:49:52.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 648540061698' 2026-02-20T16:49:52.834 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 648540061698 2026-02-20T16:49:52.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:52.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 648540061698 -lt 648540061698 2026-02-20T16:49:52.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:52.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:52.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:53.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:53.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:53.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:53.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:53.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:53.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:53.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:53.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:53.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:53.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:49:53.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-20T16:49:53.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 17 % 2 2026-02-20T16:49:53.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:49:53.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1183: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: echo ROBJ17 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1185: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:53.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:53.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:54.037 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:49:54.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:49:54.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:54.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:54.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:54.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:54.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:54.177 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:54.174+0000 7f0510829a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:54.178 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:54.178+0000 7f0510829a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:54.179 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:54.178+0000 7f0510829a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:54.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:54.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:54.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:54.551 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:54.550+0000 7f0510829a40 -1 Falling back to public interface 2026-02-20T16:49:55.199 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:55.198+0000 7f0510829a40 -1 osd.0 152 log_to_monitors true 2026-02-20T16:49:55.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:49:55.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:55.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:49:55.478 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:49:55.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:55.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 156 up_thru 156 down_at 153 last_clean_interval [146,152) [v2:127.0.0.1:6800/3772480369,v1:127.0.0.1:6801/3772480369] [v2:127.0.0.1:6802/3772480369,v1:127.0.0.1:6803/3772480369] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:49:55.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:49:55.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:49:55.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:49:55.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:49:55.635 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:49:55.635 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:49:55.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:49:55.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:49:55.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:49:55.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:49:55.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:49:55.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:49:55.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:49:55.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:49:55.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:49:55.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:49:55.908 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:49:55.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:49:55.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:55.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:49:55.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=670014898178 2026-02-20T16:49:55.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 670014898178 2026-02-20T16:49:55.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-670014898178' 2026-02-20T16:49:55.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:49:55.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:49:56.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=648540061700 2026-02-20T16:49:56.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 648540061700 2026-02-20T16:49:56.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-670014898178 1-648540061700' 2026-02-20T16:49:56.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:56.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-670014898178 2026-02-20T16:49:56.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:56.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:49:56.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-670014898178 2026-02-20T16:49:56.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:56.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=670014898178 2026-02-20T16:49:56.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 670014898178' 2026-02-20T16:49:56.059 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 670014898178 2026-02-20T16:49:56.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:56.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 627065225220 -lt 670014898178 2026-02-20T16:49:56.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:49:57.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:49:57.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:49:57.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 670014898178 -lt 670014898178 2026-02-20T16:49:57.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:49:57.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-648540061700 2026-02-20T16:49:57.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:49:57.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:49:57.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-648540061700 2026-02-20T16:49:57.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:49:57.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=648540061700 2026-02-20T16:49:57.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 648540061700' 2026-02-20T16:49:57.385 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 648540061700 2026-02-20T16:49:57.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:49:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 648540061700 -lt 648540061700 2026-02-20T16:49:57.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:49:57.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:57.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:57.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:49:57.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:49:57.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:49:57.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:49:57.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:49:57.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:49:57.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:49:57.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:49:57.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:49:57.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:49:57.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:49:57.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:49:58.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:49:58.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:49:58.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:49:58.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:49:58.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:49:58.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:49:58.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:49:58.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:49:58.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:58.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:49:58.607 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:49:58.607 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:49:58.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:49:58.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:49:58.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:58.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:49:58.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:49:58.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:49:58.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:49:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:49:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:49:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:49:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:49:58.720 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:58.718+0000 7febb322aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:58.721 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:58.722+0000 7febb322aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:58.722 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:58.722+0000 7febb322aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:49:58.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:49:58.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:49:59.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:49:59.091 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:59.090+0000 7febb322aa40 -1 Falling back to public interface 2026-02-20T16:49:59.774 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:49:59.774+0000 7febb322aa40 -1 osd.1 157 log_to_monitors true 2026-02-20T16:50:00.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:00.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:00.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:00.035 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:00.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:00.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:00.193 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [151,157) [v2:127.0.0.1:6810/1675574041,v1:127.0.0.1:6811/1675574041] [v2:127.0.0.1:6812/1675574041,v1:127.0.0.1:6813/1675574041] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:50:00.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:00.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:00.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:00.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:00.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:00.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:00.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:00.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:00.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:00.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:00.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:00.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:00.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:00.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:00.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:00.475 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:00.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:00.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:00.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=670014898180 2026-02-20T16:50:00.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 670014898180 2026-02-20T16:50:00.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-670014898180' 2026-02-20T16:50:00.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:00.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:00.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=691489734658 2026-02-20T16:50:00.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 691489734658 2026-02-20T16:50:00.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-670014898180 1-691489734658' 2026-02-20T16:50:00.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:00.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-670014898180 2026-02-20T16:50:00.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:00.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:00.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-670014898180 2026-02-20T16:50:00.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:00.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=670014898180 2026-02-20T16:50:00.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 670014898180' 2026-02-20T16:50:00.624 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 670014898180 2026-02-20T16:50:00.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:00.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 670014898178 -lt 670014898180 2026-02-20T16:50:00.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:01.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:01.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:01.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 670014898178 -lt 670014898180 2026-02-20T16:50:01.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:02.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:50:02.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 670014898180 -lt 670014898180 2026-02-20T16:50:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:03.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-691489734658 2026-02-20T16:50:03.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:03.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:03.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-691489734658 2026-02-20T16:50:03.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:03.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=691489734658 2026-02-20T16:50:03.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 691489734658' 2026-02-20T16:50:03.129 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 691489734658 2026-02-20T16:50:03.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:03.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 691489734658 -lt 691489734658 2026-02-20T16:50:03.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:03.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:03.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:03.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:03.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:03.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:03.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:03.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-20T16:50:03.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 18 % 2 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1192: TEST_corrupt_scrub_replicated: echo ROBJ18 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:50:03.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:03.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:04.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:04.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:04.345 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:50:04.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:04.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:04.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:04.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:04.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:04.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:04.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:04.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:50:04.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:50:04.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:04.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:04.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:04.445 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:04.442+0000 7fd0f0687a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:04.446 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:04.446+0000 7fd0f0687a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:04.448 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:04.446+0000 7fd0f0687a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:04.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:04.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:04.899 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:04.898+0000 7fd0f0687a40 -1 Falling back to public interface 2026-02-20T16:50:05.488 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:05.486+0000 7fd0f0687a40 -1 osd.0 162 log_to_monitors true 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:05.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:05.908 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 166 up_thru 166 down_at 163 last_clean_interval [156,162) [v2:127.0.0.1:6800/2946851191,v1:127.0.0.1:6801/2946851191] [v2:127.0.0.1:6802/2946851191,v1:127.0.0.1:6803/2946851191] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:05.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:05.910 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:05.910 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:05.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:05.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:05.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:06.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:06.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:06.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:06.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:06.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:06.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:06.187 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:06.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:06.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:06.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:06.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=712964571138 2026-02-20T16:50:06.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 712964571138 2026-02-20T16:50:06.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571138' 2026-02-20T16:50:06.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:06.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:06.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=691489734660 2026-02-20T16:50:06.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 691489734660 2026-02-20T16:50:06.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571138 1-691489734660' 2026-02-20T16:50:06.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:06.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-712964571138 2026-02-20T16:50:06.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:06.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:06.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-712964571138 2026-02-20T16:50:06.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:06.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=712964571138 2026-02-20T16:50:06.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 712964571138' 2026-02-20T16:50:06.336 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 712964571138 2026-02-20T16:50:06.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:06.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 712964571138 2026-02-20T16:50:06.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:07.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:07.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:07.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 712964571138 2026-02-20T16:50:07.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:08.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:50:08.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571138 -lt 712964571138 2026-02-20T16:50:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:08.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-691489734660 2026-02-20T16:50:08.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:08.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-691489734660 2026-02-20T16:50:08.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:08.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=691489734660 2026-02-20T16:50:08.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 691489734660' 2026-02-20T16:50:08.823 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 691489734660 2026-02-20T16:50:08.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:08.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 691489734660 -lt 691489734660 2026-02-20T16:50:08.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:08.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:08.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:09.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:09.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:09.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:09.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:09.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:09.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:09.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:09.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:09.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:09.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:09.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:09.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:09.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:09.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:09.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:09.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:50:10.022 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:50:10.022 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:10.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:50:10.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:50:10.137 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:50:10.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:10.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:50:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:50:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:10.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:10.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:10.154+0000 7fe1ea188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:10.155 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:10.154+0000 7fe1ea188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:10.156 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:10.154+0000 7fe1ea188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:10.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:10.503 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:10.502+0000 7fe1ea188a40 -1 Falling back to public interface 2026-02-20T16:50:11.106 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:11.106+0000 7fe1ea188a40 -1 osd.1 168 log_to_monitors true 2026-02-20T16:50:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:11.459 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:11.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:11.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 172 up_thru 172 down_at 169 last_clean_interval [161,168) [v2:127.0.0.1:6810/3994504782,v1:127.0.0.1:6811/3994504782] [v2:127.0.0.1:6812/3994504782,v1:127.0.0.1:6813/3994504782] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:11.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:11.623 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:11.623 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:11.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:11.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:11.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:11.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:11.897 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:11.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:11.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:11.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=712964571140 2026-02-20T16:50:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 712964571140 2026-02-20T16:50:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571140' 2026-02-20T16:50:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:11.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:12.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=738734374914 2026-02-20T16:50:12.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 738734374914 2026-02-20T16:50:12.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571140 1-738734374914' 2026-02-20T16:50:12.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:12.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-712964571140 2026-02-20T16:50:12.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:12.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-712964571140 2026-02-20T16:50:12.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:12.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=712964571140 2026-02-20T16:50:12.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 712964571140' 2026-02-20T16:50:12.047 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 712964571140 2026-02-20T16:50:12.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:12.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571138 -lt 712964571140 2026-02-20T16:50:12.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:13.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:13.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:13.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571140 -lt 712964571140 2026-02-20T16:50:13.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:13.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-738734374914 2026-02-20T16:50:13.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:13.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:13.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-738734374914 2026-02-20T16:50:13.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:13.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=738734374914 2026-02-20T16:50:13.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 738734374914' 2026-02-20T16:50:13.379 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 738734374914 2026-02-20T16:50:13.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:13.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 738734374914 -lt 738734374914 2026-02-20T16:50:13.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:13.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:13.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:13.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:13.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:13.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:13.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:14.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:14.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:14.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:14.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:14.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:14.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-20T16:50:14.560 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:50:14.561 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:14.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:14.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:14.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:50:14.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:50:14.665 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:50:14.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:14.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:50:14.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:50:14.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:14.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:14.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:14.684 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:14.682+0000 7f21a36fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:14.686 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:14.686+0000 7f21a36fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:14.687 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:14.686+0000 7f21a36fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:14.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:14.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:14.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:14.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:15.059 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:15.058+0000 7f21a36fea40 -1 Falling back to public interface 2026-02-20T16:50:15.601 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:15.598+0000 7f21a36fea40 -1 osd.1 173 log_to_monitors true 2026-02-20T16:50:15.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:15.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:15.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:15.990 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:15.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:15.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 177 up_thru 177 down_at 174 last_clean_interval [172,173) [v2:127.0.0.1:6810/1350353987,v1:127.0.0.1:6811/1350353987] [v2:127.0.0.1:6812/1350353987,v1:127.0.0.1:6813/1350353987] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:16.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:16.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:16.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:16.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:16.147 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:16.147 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:16.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:16.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:16.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:16.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:16.418 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:16.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:16.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=712964571142 2026-02-20T16:50:16.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 712964571142 2026-02-20T16:50:16.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571142' 2026-02-20T16:50:16.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:16.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:16.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=760209211394 2026-02-20T16:50:16.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 760209211394 2026-02-20T16:50:16.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571142 1-760209211394' 2026-02-20T16:50:16.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:16.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-712964571142 2026-02-20T16:50:16.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:16.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:16.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-712964571142 2026-02-20T16:50:16.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:16.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=712964571142 2026-02-20T16:50:16.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 712964571142' 2026-02-20T16:50:16.570 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 712964571142 2026-02-20T16:50:16.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:16.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571141 -lt 712964571142 2026-02-20T16:50:16.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:17.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:17.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:17.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571141 -lt 712964571142 2026-02-20T16:50:17.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:18.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:50:18.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:19.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571142 -lt 712964571142 2026-02-20T16:50:19.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:19.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-760209211394 2026-02-20T16:50:19.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:19.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:19.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-760209211394 2026-02-20T16:50:19.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:19.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=760209211394 2026-02-20T16:50:19.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2026-02-20T16:50:19.056 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 760209211394 2026-02-20T16:50:19.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:19.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 760209211394 -lt 760209211394 2026-02-20T16:50:19.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:19.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:19.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:19.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:19.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:19.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:19.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:19.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:19.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:19.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:19.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:19.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:19.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:19.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:19.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-20T16:50:19.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-20T16:50:19.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 19 % 2 2026-02-20T16:50:19.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-20T16:50:19.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-20T16:50:19.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-20T16:50:19.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-20T16:50:19.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-20T16:50:19.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:50:19.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:50:19.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-20T16:50:19.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1207: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-20T16:50:20.016 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: {} 2026-02-20T16:50:20.016 INFO:tasks.workunit.client.0.trial013.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-20T16:50:20.025 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: {} 2026-02-20T16:50:20.026 INFO:tasks.workunit.client.0.trial013.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-20T16:50:20.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1209: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:20.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-20T16:50:20.037 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:20.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:20.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.199 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.200 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:20.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:20.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:20.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:20.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:20.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:50:20.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:20.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:20.262 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.262 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.262 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:20.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:20.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:20.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:20.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-20T16:50:20.323 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-20T16:50:20.323 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:20.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:20.488 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:20.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:20.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:50:20.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:20.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:50:20.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:20.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:50:20.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:20.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.492 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.493 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:20.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:20.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:20.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:20.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.554 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:20.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:20.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-20T16:50:20.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1211: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-20T16:50:20.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:20.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:50:20.614 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:20.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:20.778 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:20.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:20.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:20.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:20.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:50:20.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:20.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:20.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:50:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:50:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:50:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:20.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:20.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:20.781 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:50:20.781 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:20.781 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:20.782 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.782 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.782 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.782 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:20.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:20.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:20.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:20.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:20.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:50:20.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:20.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:20.842 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:20.843 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:20.843 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:20.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:20.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:20.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-20T16:50:20.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-20T16:50:20.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:50:20.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:20.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:50:20.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:20.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:50:20.903 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:50:20.904 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:21.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:21.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:50:21.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:21.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:50:21.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:21.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:50:21.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:21.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:21.068 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:21.069 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:21.069 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:21.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:21.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:21.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:21.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:21.131 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:21.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:21.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:21.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-20T16:50:21.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2026-02-20T16:50:21.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:50:21.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:50:21.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:50:21.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:50:21.195 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:21.195 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:21.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:21.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:21.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:21.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:50:21.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:50:21.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:50:21.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:50:21.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:21.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:21.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:21.663 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:21.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:21.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:21.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:21.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=712964571144 2026-02-20T16:50:21.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 712964571144 2026-02-20T16:50:21.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571144' 2026-02-20T16:50:21.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:21.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:21.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=760209211396 2026-02-20T16:50:21.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 760209211396 2026-02-20T16:50:21.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-712964571144 1-760209211396' 2026-02-20T16:50:21.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:21.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-712964571144 2026-02-20T16:50:21.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:21.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:21.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-712964571144 2026-02-20T16:50:21.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:21.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=712964571144 2026-02-20T16:50:21.813 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 712964571144 2026-02-20T16:50:21.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 712964571144' 2026-02-20T16:50:21.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:21.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571142 -lt 712964571144 2026-02-20T16:50:21.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:22.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:22.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:23.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 712964571144 -lt 712964571144 2026-02-20T16:50:23.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:23.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-760209211396 2026-02-20T16:50:23.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:23.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:23.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-760209211396 2026-02-20T16:50:23.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:23.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=760209211396 2026-02-20T16:50:23.140 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 760209211396 2026-02-20T16:50:23.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 760209211396' 2026-02-20T16:50:23.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 760209211396 -lt 760209211396 2026-02-20T16:50:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:50:23.307 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:50:23.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:50:23.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:23.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:23.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:23.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:50:23.690 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:23.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:23.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:23.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:23.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:23.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:23.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:24.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:24.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:24.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:24.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:24.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:24.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:24.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:25.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:25.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:26.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:26.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:26.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:26.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:26.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:26.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:26.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:26.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:26.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:27.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:27.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:27.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:27.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:24.438220+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1216: TEST_corrupt_scrub_replicated: ERRORS=0 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1217: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1219: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2026-02-20T16:50:27.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1220: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-20T16:50:27.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1221: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-20T16:50:27.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1222: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-20T16:50:27.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1223: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-20T16:50:27.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1224: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-20T16:50:27.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1225: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1226: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1227: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1228: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1229: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1230: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1231: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1232: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1233: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1234: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1235: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1236: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-20T16:50:27.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1237: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2026-02-20T16:50:27.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1238: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2026-02-20T16:50:27.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1239: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2026-02-20T16:50:27.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:27.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:27.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1250: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-20T16:50:27.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:50:27.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-20T16:50:27.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:50:27.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-20T16:50:27.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1256: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-20T16:50:27.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:50:27.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: epoch=177 2026-02-20T16:50:27.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-20T16:50:27.588 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:50:27.588 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:50:27.589 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:50:27.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-20T16:50:27.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:50:27.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-20T16:50:27.607 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:50:27.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-20T16:50:27.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:50:27.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2174: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:50:27.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:50:27.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: test no = yes 2026-02-20T16:50:27.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2180: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-20T16:50:27.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2185: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-20T16:50:27.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2187: TEST_corrupt_scrub_replicated: echo -n ZZZ 2026-02-20T16:50:27.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-20T16:50:27.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:50:27.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:27.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:27.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:27.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:27.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:27.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:27.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:27.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:50:27.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:27.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:27.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:50:27.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:27.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:27.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:50:28.149 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:28.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:50:28.265 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:50:28.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:28.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:50:28.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:50:28.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:28.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:28.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:28.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:28.281+0000 7f06042a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:28.283 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:28.281+0000 7f06042a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:28.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:28.281+0000 7f06042a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:28.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:50:28.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:28.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:50:28.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:28.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:28.707 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:28.705+0000 7f06042a5a40 -1 Falling back to public interface 2026-02-20T16:50:29.313 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:29.309+0000 7f06042a5a40 -1 osd.0 180 log_to_monitors true 2026-02-20T16:50:29.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:29.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:29.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:29.586 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:29.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:29.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:29.745 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 183 up_thru 183 down_at 180 last_clean_interval [166,180) [v2:127.0.0.1:6800/1784838479,v1:127.0.0.1:6801/1784838479] [v2:127.0.0.1:6802/1784838479,v1:127.0.0.1:6803/1784838479] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:50:29.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:29.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:29.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:29.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:29.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:29.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:29.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:29.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:29.747 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:29.747 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:29.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:29.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:29.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:29.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:30.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:30.019 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:30.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:30.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:30.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:30.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=785979015170 2026-02-20T16:50:30.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 785979015170 2026-02-20T16:50:30.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-785979015170' 2026-02-20T16:50:30.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:30.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:30.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=760209211398 2026-02-20T16:50:30.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 760209211398 2026-02-20T16:50:30.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-785979015170 1-760209211398' 2026-02-20T16:50:30.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:30.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-785979015170 2026-02-20T16:50:30.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:30.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:30.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-785979015170 2026-02-20T16:50:30.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:30.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=785979015170 2026-02-20T16:50:30.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 785979015170' 2026-02-20T16:50:30.170 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 785979015170 2026-02-20T16:50:30.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:30.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 785979015170 -lt 785979015170 2026-02-20T16:50:30.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:30.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-760209211398 2026-02-20T16:50:30.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:30.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:30.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-760209211398 2026-02-20T16:50:30.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:30.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=760209211398 2026-02-20T16:50:30.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 760209211398' 2026-02-20T16:50:30.339 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 760209211398 2026-02-20T16:50:30.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:30.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 760209211398 -lt 760209211398 2026-02-20T16:50:30.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:30.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:30.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:30.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:30.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:30.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:30.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:30.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:30.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:30.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:31.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2191: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-20T16:50:31.049 INFO:tasks.workunit.client.0.trial013.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2193: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:31.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:50:31.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:31.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:31.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:31.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:31.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:31.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2026-02-20T16:50:31.433 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:50:31.433 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:31.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:31.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:31.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:31.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:31.547 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:31.547 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:31.547 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:31.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:31.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:31.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:50:31.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:50:31.550 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:31.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:31.568 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:31.565+0000 7fca8e587a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:31.569 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:31.569+0000 7fca8e587a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:31.570 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:31.569+0000 7fca8e587a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:31.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:50:31.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:31.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:31.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:31.967 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:31.965+0000 7fca8e587a40 -1 Falling back to public interface 2026-02-20T16:50:32.588 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:32.585+0000 7fca8e587a40 -1 osd.1 185 log_to_monitors true 2026-02-20T16:50:32.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:32.865 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 188 up_thru 188 down_at 185 last_clean_interval [177,185) [v2:127.0.0.1:6810/3068730459,v1:127.0.0.1:6811/3068730459] [v2:127.0.0.1:6812/3068730459,v1:127.0.0.1:6813/3068730459] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:33.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:33.026 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:33.027 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:33.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:33.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:33.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:33.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:33.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:33.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:33.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:33.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:33.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:33.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:33.291 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:33.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:33.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:33.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:33.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=785979015171 2026-02-20T16:50:33.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 785979015171 2026-02-20T16:50:33.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-785979015171' 2026-02-20T16:50:33.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:33.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=807453851650 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 807453851650 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-785979015171 1-807453851650' 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-785979015171 2026-02-20T16:50:33.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:33.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:33.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-785979015171 2026-02-20T16:50:33.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=785979015171 2026-02-20T16:50:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 785979015171' 2026-02-20T16:50:33.442 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 785979015171 2026-02-20T16:50:33.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 785979015170 -lt 785979015171 2026-02-20T16:50:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:34.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:34.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:34.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 785979015171 -lt 785979015171 2026-02-20T16:50:34.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:34.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-807453851650 2026-02-20T16:50:34.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:34.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:34.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-807453851650 2026-02-20T16:50:34.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=807453851650 2026-02-20T16:50:34.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 807453851650' 2026-02-20T16:50:34.770 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 807453851650 2026-02-20T16:50:34.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 807453851650 -lt 807453851650 2026-02-20T16:50:34.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:35.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:35.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:35.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:35.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:35.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:35.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:35.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2195: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:50:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:35.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:50:35.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:35.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:35.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:35.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:35.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:35.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:35.997 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:50:36.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:36.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:36.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:36.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:36.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:36.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:36.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:36.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:50:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:50:36.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:36.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:36.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:36.125 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:36.125+0000 7faa91d9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:36.127 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:36.125+0000 7faa91d9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:36.128 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:36.125+0000 7faa91d9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:36.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:50:36.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:36.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:50:36.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:36.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:36.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:36.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:36.272 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:36.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:36.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:36.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:36.511 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:36.509+0000 7faa91d9ea40 -1 Falling back to public interface 2026-02-20T16:50:37.085 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:37.081+0000 7faa91d9ea40 -1 osd.0 189 log_to_monitors true 2026-02-20T16:50:37.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:37.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:37.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:37.437 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:37.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:37.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:50:37.593 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 193 up_thru 193 down_at 190 last_clean_interval [183,189) [v2:127.0.0.1:6800/117764338,v1:127.0.0.1:6801/117764338] [v2:127.0.0.1:6802/117764338,v1:127.0.0.1:6803/117764338] exists,up 424c99bd-55ea-47a7-b5e9-87625992f8bd 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:37.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:37.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:37.595 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:37.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:37.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:37.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:37.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:37.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:37.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:37.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:37.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:37.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:37.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:37.876 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:37.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:37.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:37.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:37.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=828928688130 2026-02-20T16:50:37.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 828928688130 2026-02-20T16:50:37.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688130' 2026-02-20T16:50:37.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:37.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:38.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=807453851652 2026-02-20T16:50:38.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 807453851652 2026-02-20T16:50:38.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688130 1-807453851652' 2026-02-20T16:50:38.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:38.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-828928688130 2026-02-20T16:50:38.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:38.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:38.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-828928688130 2026-02-20T16:50:38.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:38.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=828928688130 2026-02-20T16:50:38.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 828928688130' 2026-02-20T16:50:38.026 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 828928688130 2026-02-20T16:50:38.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:38.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 785979015172 -lt 828928688130 2026-02-20T16:50:38.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:39.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:39.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:39.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688130 -lt 828928688130 2026-02-20T16:50:39.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:39.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-807453851652 2026-02-20T16:50:39.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:39.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:39.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-807453851652 2026-02-20T16:50:39.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:39.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=807453851652 2026-02-20T16:50:39.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 807453851652' 2026-02-20T16:50:39.351 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 807453851652 2026-02-20T16:50:39.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:39.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 807453851652 -lt 807453851652 2026-02-20T16:50:39.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:39.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:39.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:39.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:39.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:39.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:39.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:39.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:39.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:39.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:39.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:39.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:39.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:39.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:39.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:40.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:40.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2197: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:50:40.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:50:40.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:50:40.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:50:40.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:50:40.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:50:40.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:50:40.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:40.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-20T16:50:40.565 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:50:40.566 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:50:40.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:50:40.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:50:40.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:50:40.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:50:40.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:50:40.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:50:40.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:50:40.683 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:40.681+0000 7f4c1a738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:40.684 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:40.681+0000 7f4c1a738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:40.685 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:40.685+0000 7f4c1a738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:50:40.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:40.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:40.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:50:41.071 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:41.069+0000 7f4c1a738a40 -1 Falling back to public interface 2026-02-20T16:50:41.645 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:50:41.641+0000 7f4c1a738a40 -1 osd.1 195 log_to_monitors true 2026-02-20T16:50:41.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:50:41.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:50:41.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:50:41.981 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:50:41.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:50:41.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:50:42.142 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 199 up_thru 199 down_at 196 last_clean_interval [188,195) [v2:127.0.0.1:6810/334616322,v1:127.0.0.1:6811/334616322] [v2:127.0.0.1:6812/334616322,v1:127.0.0.1:6813/334616322] exists,up d1dd27eb-1c02-4ef5-896c-f6aa115eec8d 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:42.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:42.144 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:42.145 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:42.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:42.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:42.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:42.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:42.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:42.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:42.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:42.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:42.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:42.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:42.424 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:42.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:42.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:42.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:42.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=828928688132 2026-02-20T16:50:42.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 828928688132 2026-02-20T16:50:42.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688132' 2026-02-20T16:50:42.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:42.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:42.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=854698491906 2026-02-20T16:50:42.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 854698491906 2026-02-20T16:50:42.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688132 1-854698491906' 2026-02-20T16:50:42.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:42.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-828928688132 2026-02-20T16:50:42.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:42.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:42.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-828928688132 2026-02-20T16:50:42.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:42.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=828928688132 2026-02-20T16:50:42.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 828928688132' 2026-02-20T16:50:42.576 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 828928688132 2026-02-20T16:50:42.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:42.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688131 -lt 828928688132 2026-02-20T16:50:42.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:43.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:43.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:43.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688131 -lt 828928688132 2026-02-20T16:50:43.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:44.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:50:44.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:45.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688132 -lt 828928688132 2026-02-20T16:50:45.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:45.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-854698491906 2026-02-20T16:50:45.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:45.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:45.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-854698491906 2026-02-20T16:50:45.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:45.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=854698491906 2026-02-20T16:50:45.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 854698491906' 2026-02-20T16:50:45.066 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 854698491906 2026-02-20T16:50:45.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:45.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 854698491906 -lt 854698491906 2026-02-20T16:50:45.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:50:45.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:45.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:45.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:50:45.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:50:45.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:50:45.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:50:45.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:50:45.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:50:45.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:50:45.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:50:45.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:50:45.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:50:45.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:50:45.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:50:45.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:50:45.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:50:45.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:50:45.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2198: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2026-02-20T16:50:45.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2200: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-20T16:50:45.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:45.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:45.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:50:45.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-20T16:50:45.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-20T16:50:45.775 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:45.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:45.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:50:45.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:45.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:45.940 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:45.940 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:45.941 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:45.941 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:45.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:46.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:46.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:46.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:46.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:50:46.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:46.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:46.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2201: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:46.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-20T16:50:46.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-20T16:50:46.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:50:46.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:46.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:46.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:46.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:46.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:46.228 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:50:46.228 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:46.228 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:46.229 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.229 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.229 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:46.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:46.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:46.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:46.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:46.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-20T16:50:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2202: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-20T16:50:46.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:50:46.351 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:50:46.352 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:46.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:46.517 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:46.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:46.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:46.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:46.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:50:46.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:46.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:46.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:50:46.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:46.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:50:46.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:46.520 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:46.521 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.521 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.521 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.521 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:46.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:46.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:46.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:50:46.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:50:46.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:50:46.581 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.582 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.582 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:50:46.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:46.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2203: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:50:46.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:50:46.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:50:46.643 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:50:46.643 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:50:46.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:50:46.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:50:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:50:46.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:50:46.807 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:46.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:50:46.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:50:46.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:50:46.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:50:46.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:50:46.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:50:46.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:50:46.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-20T16:50:46.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2206: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2026-02-20T16:50:46.997 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: {} 2026-02-20T16:50:46.998 INFO:tasks.workunit.client.0.trial013.stderr:osd.0: osd_max_object_size = '' 2026-02-20T16:50:47.007 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: {} 2026-02-20T16:50:47.007 INFO:tasks.workunit.client.0.trial013.stderr:osd.1: osd_max_object_size = '' 2026-02-20T16:50:47.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2208: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-20T16:50:47.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:50:47.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:50:47.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:50:47.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:50:47.016 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:47.016 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:50:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:50:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:50:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:50:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:47.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:47.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:47.475 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:47.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:47.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:47.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:47.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=828928688134 2026-02-20T16:50:47.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 828928688134 2026-02-20T16:50:47.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688134' 2026-02-20T16:50:47.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:47.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:47.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=854698491908 2026-02-20T16:50:47.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 854698491908 2026-02-20T16:50:47.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688134 1-854698491908' 2026-02-20T16:50:47.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:47.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-828928688134 2026-02-20T16:50:47.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:47.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:47.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-828928688134 2026-02-20T16:50:47.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:47.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=828928688134 2026-02-20T16:50:47.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 828928688134' 2026-02-20T16:50:47.628 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 828928688134 2026-02-20T16:50:47.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:47.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688132 -lt 828928688134 2026-02-20T16:50:47.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:50:48.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:50:48.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:48.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688134 -lt 828928688134 2026-02-20T16:50:48.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:48.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-854698491908 2026-02-20T16:50:48.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:48.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:50:48.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-854698491908 2026-02-20T16:50:48.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:48.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=854698491908 2026-02-20T16:50:48.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 854698491908' 2026-02-20T16:50:48.964 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 854698491908 2026-02-20T16:50:48.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:50:49.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 854698491908 -lt 854698491908 2026-02-20T16:50:49.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:50:49.132 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:50:49.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:50:49.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:50:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:50:49.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:50:49.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:50:49.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:50:49.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:50:49.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:50:49.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:50:49.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:50:49.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:50:49.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:49.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:50:49.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:49.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:50:49.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:49.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:50:49.510 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:48:19.669396+0000 last_deep_scrub_stamp 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:49.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:50:49.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:49.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:50:49.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:49.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:50.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:50.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:50.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:50:50.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:50.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:50:50.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:50.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:50:50.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:50.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:51.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:51.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:50:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:50:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:50:52.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:48:19.669396+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:52.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:53.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:53.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:53.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:50:53.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:53.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:50:53.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:53.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:50:53.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:49.566100+0000 '>' 2026-02-20T16:48:19.669396+0000 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2210: TEST_corrupt_scrub_replicated: err_strings=() 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2211: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2212: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2213: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2214: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2215: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-20T16:50:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2216: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2217: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2218: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2219: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2220: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2221: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2222: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2223: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2224: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2225: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2226: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2227: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2228: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2229: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2230: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2231: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2232: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2233: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2234: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2235: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2236: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2237: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2026-02-20T16:50:53.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2238: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2239: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2240: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2241: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2242: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2243: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2244: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2245: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2246: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2247: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2248: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2249: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2026-02-20T16:50:53.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-20T16:50:53.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2026-02-20T16:50:53.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2260: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-20T16:50:53.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:50:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-20T16:50:53.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:50:53.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-20T16:50:53.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2266: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-20T16:50:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-20T16:50:53.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: epoch=199 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:50:53.548 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-20T16:50:53.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:50:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-20T16:50:53.572 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:50:53.573 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:50:53.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-20T16:50:53.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:50:53.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:50:53.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3700: TEST_corrupt_scrub_replicated: test no = yes 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3710: TEST_corrupt_scrub_replicated: repair 3.0 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=3.0 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 3.0 2026-02-20T16:50:53.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:53.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:53.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:53.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:53.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:53.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 3.0 2026-02-20T16:50:53.905 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to repair 2026-02-20T16:50:53.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 3.0 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:53.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:50:53.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:53.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:53.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:53.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:54.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:49.566100+0000 '>' 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:54.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:55.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:55.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:55.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:55.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:55.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:55.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:55.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:55.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:49.566100+0000 '>' 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:55.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:56.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:56.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:56.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:56.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:56.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:56.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:56.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:56.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:49.566100+0000 '>' 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:56.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:57.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:57.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:57.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:57.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:57.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:57.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:57.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:57.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:49.566100+0000 '>' 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:57.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:50:58.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:50:58.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:50:58.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:50:58.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:50:58.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:50:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:50:58.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:50:55.505832+0000 '>' 2026-02-20T16:50:49.566100+0000 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:50:58.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:50:58.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:50:58.727 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:50:58.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:50:58.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:50:58.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:50:58.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:50:58.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:50:58.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:50:58.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:50:58.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:50:58.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:50:59.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:50:59.002 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:50:59.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:50:59.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:59.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:50:59.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=828928688137 2026-02-20T16:50:59.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 828928688137 2026-02-20T16:50:59.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688137' 2026-02-20T16:50:59.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:50:59.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:50:59.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=854698491911 2026-02-20T16:50:59.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 854698491911 2026-02-20T16:50:59.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-828928688137 1-854698491911' 2026-02-20T16:50:59.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:50:59.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-828928688137 2026-02-20T16:50:59.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:50:59.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:50:59.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-828928688137 2026-02-20T16:50:59.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:50:59.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=828928688137 2026-02-20T16:50:59.149 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 828928688137 2026-02-20T16:50:59.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 828928688137' 2026-02-20T16:50:59.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:50:59.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688136 -lt 828928688137 2026-02-20T16:50:59.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:00.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:00.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:00.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 828928688137 -lt 828928688137 2026-02-20T16:51:00.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:00.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-854698491911 2026-02-20T16:51:00.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:00.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:00.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-854698491911 2026-02-20T16:51:00.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=854698491911 2026-02-20T16:51:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 854698491911' 2026-02-20T16:51:00.477 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 854698491911 2026-02-20T16:51:00.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:00.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 854698491911 -lt 854698491911 2026-02-20T16:51:00.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:00.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:00.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:00.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:51:00.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:00.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:00.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:00.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:00.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:00.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:00.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:00.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:51:00.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:00.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:00.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:01.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:51:01.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:01.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:01.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3714: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-20T16:51:01.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3715: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-20T16:51:01.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3717: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-20T16:51:01.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3718: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-20T16:51:01.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3719: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-20T16:51:01.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3720: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-20T16:51:01.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3722: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2026-02-20T16:51:01.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3728: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-20T16:51:01.432 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' removed 2026-02-20T16:51:01.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:51:01.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:51:01.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:51:01.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:01.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:01.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:01.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:01.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:01.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:01.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:51:01.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:51:01.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:51:01.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:51:01.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:51:01.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:51:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:51:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:51:01.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:01.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:51:01.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:51:01.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:51:01.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:51:01.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:01.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:01.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:01.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:51:01.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:51:01.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:51:01.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:51:01.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:51:01.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:51:01.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:01.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:51:01.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:51:01.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:01.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:51:01.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:51:01.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:51:01.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:51:01.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:51:01.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:51:01.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:51:01.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:51:01.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:51:01.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:51:01.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6100: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6101: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2026-02-20T16:51:01.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6102: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6104: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:51:01.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.847 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:01.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T16:51:01.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:51:01.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:51:01.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:51:01.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:51:01.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:51:01.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:51:01.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:51:01.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:51:01.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:51:01.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:51:01.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:51:01.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:51:01.938 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:01.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:51:01.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:51:01.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:51:01.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6105: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:51:01.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:51:02.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:51:02.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:02.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:02.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:02.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:02.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:02.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:02.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:51:02.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:51:02.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6106: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:51:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:02.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:51:02.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:51:02.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:51:02.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3f18c1cd-578d-4203-9cb3-5648b44fd981 2026-02-20T16:51:02.127 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 3f18c1cd-578d-4203-9cb3-5648b44fd981 2026-02-20T16:51:02.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3f18c1cd-578d-4203-9cb3-5648b44fd981' 2026-02-20T16:51:02.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:51:02.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2kJhpAzsyCBAAA+nSWCyCY9zV3H31lvtgaw== 2026-02-20T16:51:02.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2kJhpAzsyCBAAA+nSWCyCY9zV3H31lvtgaw=="}' 2026-02-20T16:51:02.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3f18c1cd-578d-4203-9cb3-5648b44fd981 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:51:02.224 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:02.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:51:02.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2kJhpAzsyCBAAA+nSWCyCY9zV3H31lvtgaw== --osd-uuid 3f18c1cd-578d-4203-9cb3-5648b44fd981 2026-02-20T16:51:02.250 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:02.249+0000 7f79b97a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:02.252 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:02.249+0000 7f79b97a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:02.253 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:02.249+0000 7f79b97a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:02.253 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:02.253+0000 7f79b97a7a40 -1 bdev(0x55bae5d74c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:51:02.254 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:02.253+0000 7f79b97a7a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:51:02.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:51:02.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:51:02.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:51:02.947 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:51:02.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:51:03.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:51:03.041 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:51:03.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:51:03.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:51:03.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:51:03.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:51:03.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:03.053+0000 7fc699932a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:03.057 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:03.053+0000 7fc699932a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:03.058 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:03.057+0000 7fc699932a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:03.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:03.135 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:03.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:03.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:03.495 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:03.493+0000 7fc699932a40 -1 Falling back to public interface 2026-02-20T16:51:04.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:04.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:04.229 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:04.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:04.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:04.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:04.238 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:04.237+0000 7fc699932a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:51:04.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:05.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:05.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:05.360 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:51:05.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:51:05.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:05.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:05.519 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3112206068,v1:127.0.0.1:6801/3112206068] [v2:127.0.0.1:6802/3112206068,v1:127.0.0.1:6803/3112206068] exists,up 3f18c1cd-578d-4203-9cb3-5648b44fd981 2026-02-20T16:51:05.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:05.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:05.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:05.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6107: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2026-02-20T16:51:05.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:51:05.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:51:05.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:51:05.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:51:05.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db6b6f0b-736c-4220-9491-766ac1dc9b34 2026-02-20T16:51:05.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 db6b6f0b-736c-4220-9491-766ac1dc9b34' 2026-02-20T16:51:05.527 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 db6b6f0b-736c-4220-9491-766ac1dc9b34 2026-02-20T16:51:05.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:51:05.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5kJhpMOsbIBAAv0wvDi6f/ol58+fUm9j5jg== 2026-02-20T16:51:05.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5kJhpMOsbIBAAv0wvDi6f/ol58+fUm9j5jg=="}' 2026-02-20T16:51:05.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db6b6f0b-736c-4220-9491-766ac1dc9b34 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:51:05.701 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:05.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:51:05.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5kJhpMOsbIBAAv0wvDi6f/ol58+fUm9j5jg== --osd-uuid db6b6f0b-736c-4220-9491-766ac1dc9b34 2026-02-20T16:51:05.737 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:05.733+0000 7faa005e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:05.738 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:05.737+0000 7faa005e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:05.739 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:05.737+0000 7faa005e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:05.740 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:05.737+0000 7faa005e1a40 -1 bdev(0x5560be630c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:51:05.740 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:05.737+0000 7faa005e1a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:51:06.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:51:06.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:51:06.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:51:06.398 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:51:06.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:51:06.606 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:51:06.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:51:06.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:51:06.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:51:06.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:51:06.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:51:06.629 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:06.625+0000 7f0001936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:06.631 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:06.629+0000 7f0001936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:06.632 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:06.629+0000 7f0001936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:06.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:06.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:07.047 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:07.045+0000 7f0001936a40 -1 Falling back to public interface 2026-02-20T16:51:07.625 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:07.620+0000 7f0001936a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:51:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:07.928 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:07.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:07.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:07.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:08.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:09.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:09.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:09.089 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:51:09.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:51:09.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:09.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:09.249 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/845112534,v1:127.0.0.1:6811/845112534] [v2:127.0.0.1:6812/845112534,v1:127.0.0.1:6813/845112534] exists,up db6b6f0b-736c-4220-9491-766ac1dc9b34 2026-02-20T16:51:09.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:09.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:09.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:09.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6108: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2026-02-20T16:51:09.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:51:09.399 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:51:09.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:51:09.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:51:09.622 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:51:09.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:10.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:51:10.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6109: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-20T16:51:10.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:10.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:10.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:10.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:10.915 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:10.915 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:10.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:10.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:10.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:11.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:11.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:11.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:11.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:11.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:11.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:11.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:11.192 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:11.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:11.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:11.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:51:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:51:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:51:11.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:11.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-20T16:51:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-20T16:51:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-20T16:51:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:11.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:51:11.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:11.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:11.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:51:11.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:11.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:51:11.342 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:51:11.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:51:11.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:11.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T16:51:11.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:12.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:12.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:12.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:51:12.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:12.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-20T16:51:12.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:12.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:12.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-20T16:51:12.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:12.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-20T16:51:12.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-20T16:51:12.673 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672962 2026-02-20T16:51:12.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:12.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-20T16:51:12.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:12.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:12.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:13.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:51:13.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:13.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:13.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:13.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:13.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:13.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:13.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:13.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:51:13.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:13.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:13.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:13.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:51:13.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:13.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:13.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6111: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2026-02-20T16:51:13.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-20T16:51:13.607 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T16:51:13.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:14.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6112: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2026-02-20T16:51:14.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-20T16:51:14.826 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' created 2026-02-20T16:51:14.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6113: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-20T16:51:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:15.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:15.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:15.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:15.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:15.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:15.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:15.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:16.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:16.120 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:16.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:16.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:16.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:16.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:51:16.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:51:16.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:51:16.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:16.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-20T16:51:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-20T16:51:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-20T16:51:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:16.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:51:16.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:16.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:16.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:51:16.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:51:16.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:51:16.271 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:51:16.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:16.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-20T16:51:16.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:17.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:17.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:51:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:17.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-20T16:51:17.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:17.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:17.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-20T16:51:17.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:17.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-20T16:51:17.597 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672964 2026-02-20T16:51:17.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-20T16:51:17.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-20T16:51:17.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:17.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:17.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:51:17.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:17.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:17.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:17.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:17.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:17.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:17.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:18.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:51:18.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:18.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:18.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:18.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:51:18.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:18.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:18.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-20T16:51:18.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-20T16:51:18.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:18.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:18.556 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:18.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:18.756 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:18.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:18.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:18.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:18.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-20T16:51:18.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:18.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:19.051 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:19.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:19.255 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:19.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:19.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-20T16:51:19.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-20T16:51:19.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2026-02-20T16:51:19.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-20T16:51:19.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-20T16:51:19.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:51:19.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:51:19.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2026-02-20T16:51:19.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2026-02-20T16:51:19.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-20T16:51:19.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-20T16:51:19.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:51:19.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:51:19.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: local primary=1 2026-02-20T16:51:19.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6126: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2026-02-20T16:51:19.724 INFO:tasks.workunit.client.0.trial013.stdout:created pool csr_pool snap snap1 2026-02-20T16:51:19.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6127: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2026-02-20T16:51:19.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-20T16:51:19.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-20T16:51:19.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-20T16:51:19.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2026-02-20T16:51:19.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=1 2026-02-20T16:51:19.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-20T16:51:19.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6137: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6138: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:51:19.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:51:19.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:19.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:19.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:19.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:19.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:20.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:20.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-20T16:51:20.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:51:20.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:51:20.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:51:20.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:51:20.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:51:20.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:20.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:20.343 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:20.343 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:20.343 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:20.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:51:20.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:51:20.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:51:20.345 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:51:20.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:51:20.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:51:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:51:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:51:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:51:20.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:51:20.364 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:20.360+0000 7ffb81fdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:20.366 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:20.364+0000 7ffb81fdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:20.367 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:20.364+0000 7ffb81fdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:20.510 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:51:20.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:20.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:20.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:20.727 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:20.724+0000 7ffb81fdca40 -1 Falling back to public interface 2026-02-20T16:51:21.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:21.312+0000 7ffb81fdca40 -1 osd.1 29 log_to_monitors true 2026-02-20T16:51:21.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:21.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:21.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:21.667 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:51:21.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:21.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:21.826 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/625324992,v1:127.0.0.1:6811/625324992] [v2:127.0.0.1:6812/625324992,v1:127.0.0.1:6813/625324992] exists,up db6b6f0b-736c-4220-9491-766ac1dc9b34 2026-02-20T16:51:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:51:21.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:21.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:21.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:21.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:21.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:22.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:22.104 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:22.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:22.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:22.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:22.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-20T16:51:22.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-20T16:51:22.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-20T16:51:22.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:22.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:22.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920770 2026-02-20T16:51:22.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920770 2026-02-20T16:51:22.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-141733920770' 2026-02-20T16:51:22.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:22.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-20T16:51:22.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:22.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:22.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-20T16:51:22.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:22.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-20T16:51:22.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-20T16:51:22.256 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836487 2026-02-20T16:51:22.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:22.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-20T16:51:22.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:23.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:23.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:23.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-20T16:51:23.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:23.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920770 2026-02-20T16:51:23.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:23.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:23.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920770 2026-02-20T16:51:23.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:23.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920770 2026-02-20T16:51:23.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2026-02-20T16:51:23.592 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 141733920770 2026-02-20T16:51:23.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:23.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920770 -lt 141733920770 2026-02-20T16:51:23.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:23.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:23.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:51:23.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:23.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:23.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:24.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:51:24.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:24.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:24.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-20T16:51:24.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2026-02-20T16:51:24.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=0 2026-02-20T16:51:24.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-20T16:51:24.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6142: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2026-02-20T16:51:24.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6143: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:51:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:51:24.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:24.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:24.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:24.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:24.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:51:24.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:51:24.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2026-02-20T16:51:24.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:51:24.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:51:24.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:24.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:51:24.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:24.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:24.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:51:24.903 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:51:24.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:51:24.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:51:24.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:51:24.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:51:24.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:51:24.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:51:24.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:24.916+0000 7f5972ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:24.922 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:24.920+0000 7f5972ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:24.923 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:24.920+0000 7f5972ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:25.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:25.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:25.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:25.344+0000 7f5972ba6a40 -1 Falling back to public interface 2026-02-20T16:51:25.930 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:25.928+0000 7f5972ba6a40 -1 osd.0 35 log_to_monitors true 2026-02-20T16:51:26.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:26.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:26.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:26.228 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:51:26.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:26.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:26.381 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6800/671321335,v1:127.0.0.1:6801/671321335] [v2:127.0.0.1:6802/671321335,v1:127.0.0.1:6803/671321335] exists,up 3f18c1cd-578d-4203-9cb3-5648b44fd981 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:26.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:26.383 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:26.383 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:26.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:26.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:26.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:26.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:26.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:26.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:26.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:26.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:26.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:26.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:26.672 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:26.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:26.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:26.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-20T16:51:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-20T16:51:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724546' 2026-02-20T16:51:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:26.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:26.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920772 2026-02-20T16:51:26.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920772 2026-02-20T16:51:26.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724546 1-141733920772' 2026-02-20T16:51:26.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:26.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-167503724546 2026-02-20T16:51:26.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:26.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-167503724546 2026-02-20T16:51:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:26.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-20T16:51:26.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2026-02-20T16:51:26.828 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 167503724546 2026-02-20T16:51:26.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:26.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 167503724546 2026-02-20T16:51:26.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:27.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:27.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:28.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 167503724546 2026-02-20T16:51:28.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:29.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:51:29.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-20T16:51:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:29.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920772 2026-02-20T16:51:29.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:29.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:29.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920772 2026-02-20T16:51:29.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:29.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920772 2026-02-20T16:51:29.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920772' 2026-02-20T16:51:29.326 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 141733920772 2026-02-20T16:51:29.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:29.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920772 -lt 141733920772 2026-02-20T16:51:29.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:29.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:29.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:29.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:29.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:51:29.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:29.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:29.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:30.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:51:30.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:30.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:30.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6148: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2026-02-20T16:51:30.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6150: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2026-02-20T16:51:30.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:51:30.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:51:30.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:51:30.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:51:30.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:30.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:30.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:30.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:30.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:30.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:30.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:30.492 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:30.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:30.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:30.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:30.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724547 2026-02-20T16:51:30.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724547 2026-02-20T16:51:30.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547' 2026-02-20T16:51:30.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:30.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:30.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920773 2026-02-20T16:51:30.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920773 2026-02-20T16:51:30.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-141733920773' 2026-02-20T16:51:30.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:30.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-167503724547 2026-02-20T16:51:30.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:30.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:30.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-167503724547 2026-02-20T16:51:30.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:30.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724547 2026-02-20T16:51:30.641 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 167503724547 2026-02-20T16:51:30.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2026-02-20T16:51:30.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:30.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724547 2026-02-20T16:51:30.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:31.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:31.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:31.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724547 2026-02-20T16:51:31.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:32.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:51:32.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:33.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724548 -lt 167503724547 2026-02-20T16:51:33.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:33.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920773 2026-02-20T16:51:33.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:33.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:33.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920773 2026-02-20T16:51:33.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:33.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920773 2026-02-20T16:51:33.141 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 141733920773 2026-02-20T16:51:33.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2026-02-20T16:51:33.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:33.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920774 -lt 141733920773 2026-02-20T16:51:33.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:51:33.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:51:33.312 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:51:33.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:51:33.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:51:33.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:51:33.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:51:33.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:51:33.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:51:33.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:33.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:33.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:33.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:51:33.685 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:51:33.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:51:33.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:51:33.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:33.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:33.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:33.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:33.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:14.827875+0000 '>' 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:33.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:51:34.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:51:34.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:51:34.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:51:34.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:34.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:34.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:34.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:35.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:14.827875+0000 '>' 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:35.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:51:36.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:51:36.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:51:36.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:51:36.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:36.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:36.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:36.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:36.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:14.827875+0000 '>' 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:36.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:51:37.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:51:37.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:51:37.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:51:37.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:37.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:37.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:37.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:37.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:14.827875+0000 '>' 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:37.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:51:38.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:51:38.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:51:38.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:51:38.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:51:38.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:51:38.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:51:38.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:51:38.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:34.167847+0000 '>' 2026-02-20T16:51:14.827875+0000 2026-02-20T16:51:38.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:51:38.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6152: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2026-02-20T16:51:38.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2026-02-20T16:51:38.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2026-02-20T16:51:38.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-20T16:51:38.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2026-02-20T16:51:38.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:51:38.561 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-20T16:51:38.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:51:38.581 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:51:38.582 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-20T16:51:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-20T16:51:38.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:51:38.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6330: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:51:38.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-20T16:51:38.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6331: TEST_corrupt_snapset_scrub_rep: test no = yes 2026-02-20T16:51:38.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6336: TEST_corrupt_snapset_scrub_rep: test '' = yes 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6341: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6342: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6343: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6344: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6345: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6346: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6347: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6348: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-20T16:51:38.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2026-02-20T16:51:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6359: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2026-02-20T16:51:38.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6365: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-20T16:51:38.827 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' removed 2026-02-20T16:51:38.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:51:38.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:51:38.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:51:38.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:51:38.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:38.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:38.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:38.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:38.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:38.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:38.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:51:38.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:51:38.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:51:38.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:51:38.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:51:38.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:51:38.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:38.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:51:38.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:51:38.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:38.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:51:38.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:51:38.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:51:38.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:51:38.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:38.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:38.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:51:38.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:51:38.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:51:38.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:51:38.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:51:38.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:51:38.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:51:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:51:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:51:38.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:51:38.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:51:38.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:51:38.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:51:38.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:51:38.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:51:38.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:51:38.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:51:38.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:38.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:51:38.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:51:38.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:51:38.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:51:38.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:51:38.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:51:38.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:51:38.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:38.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:38.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:51:38.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:51:38.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:51:38.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:51:38.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:51:38.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:38.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:38.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:51:38.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:51:38.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:51:38.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:51:38.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:51:38.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_dual_store_replicated_cluster td/osd-scrub-repair 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6432: TEST_dual_store_replicated_cluster: local dir=td/osd-scrub-repair 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6433: TEST_dual_store_replicated_cluster: local poolname=csr_pool 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6434: TEST_dual_store_replicated_cluster: local total_objs=19 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6435: TEST_dual_store_replicated_cluster: local extr_dbg=1 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6437: TEST_dual_store_replicated_cluster: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:51:38.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-20T16:51:39.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:51:39.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:39.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:39.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:39.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:39.018 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:39.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:39.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T16:51:39.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:51:39.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:51:39.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:51:39.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:51:39.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:51:39.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:51:39.046 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:51:39.047 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:51:39.047 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:51:39.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:51:39.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:51:39.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:51:39.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:51:39.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:51:39.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:51:39.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:51:39.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:51:39.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:51:39.108 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:51:39.109 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:39.109 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:39.109 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:51:39.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:51:39.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6438: TEST_dual_store_replicated_cluster: run_mgr td/osd-scrub-repair x --mgr_stats_period=1 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:51:39.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:51:39.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:51:39.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:39.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:39.257 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:39.257 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:39.257 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:39.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:39.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:51:39.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T16:51:39.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6439: TEST_dual_store_replicated_cluster: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-20T16:51:39.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6440: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-20T16:51:39.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6441: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 ' 2026-02-20T16:51:39.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: seq 0 1 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:51:39.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:39.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:39.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:39.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:39.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:51:39.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:51:39.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:51:39.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:51:39.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:51:39.286 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:51:39.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04816cab-d675-45b4-8f5b-ca3ee92c34e1' 2026-02-20T16:51:39.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:51:39.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbkZhpLDKwERAALj2KrrpnjHILoAXiFMviBQ== 2026-02-20T16:51:39.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbkZhpLDKwERAALj2KrrpnjHILoAXiFMviBQ=="}' 2026-02-20T16:51:39.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04816cab-d675-45b4-8f5b-ca3ee92c34e1 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:51:39.386 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:51:39.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQAbkZhpLDKwERAALj2KrrpnjHILoAXiFMviBQ== --osd-uuid 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:51:39.414 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:39.412+0000 7fc4848dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:39.415 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:39.412+0000 7fc4848dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:39.416 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:39.412+0000 7fc4848dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:39.416 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:39.412+0000 7fc4848dea40 -1 bdev(0x5608d1e78c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:51:39.417 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:39.412+0000 7fc4848dea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:51:40.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:51:40.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:51:40.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:51:40.247 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:51:40.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:51:40.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:51:40.351 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:51:40.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:51:40.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:51:40.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:51:40.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:51:40.379 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:40.376+0000 7f61fbb5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:40.381 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:40.376+0000 7f61fbb5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:40.382 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:40.380+0000 7f61fbb5ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:40.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:51:40.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:40.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:51:40.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:40.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:40.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:40.449 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:40.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:40.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:40.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:40.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:40.827 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:40.824+0000 7f61fbb5ca40 -1 Falling back to public interface 2026-02-20T16:51:41.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:41.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:41.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:41.543 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:41.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:41.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:41.578 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:41.576+0000 7f61fbb5ca40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:51:41.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:42.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:42.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:42.641 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:51:42.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:51:42.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:42.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:51:42.799 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2325989523,v1:127.0.0.1:6801/2325989523] [v2:127.0.0.1:6802/2325989523,v1:127.0.0.1:6803/2325989523] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:51:42.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:51:42.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:51:42.804 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:51:42.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:51:42.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:51:42.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:51:42.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:51:42.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:51:42.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:51:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:51:42.807 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:51:42.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2efb251e-85ef-4aa0-8760-d56d550aa12b' 2026-02-20T16:51:42.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:51:42.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAekZhpVe3HMBAA5XNJ1DkCU4NsM4KYNOZg+g== 2026-02-20T16:51:42.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAekZhpVe3HMBAA5XNJ1DkCU4NsM4KYNOZg+g=="}' 2026-02-20T16:51:42.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2efb251e-85ef-4aa0-8760-d56d550aa12b -i td/osd-scrub-repair/1/new.json 2026-02-20T16:51:42.977 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:42.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:51:42.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQAekZhpVe3HMBAA5XNJ1DkCU4NsM4KYNOZg+g== --osd-uuid 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:51:43.005 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.000+0000 7f80e3b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:43.006 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.004+0000 7f80e3b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:43.007 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.004+0000 7f80e3b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:43.008 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.004+0000 7f80e3b3ca40 -1 bdev(0x55d18969cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:51:43.008 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.004+0000 7f80e3b3ca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:51:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:51:43.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:51:43.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:51:43.669 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:51:43.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:51:43.871 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:51:43.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:51:43.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:51:43.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:51:43.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:51:43.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:51:43.897 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.892+0000 7f58bd9b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:43.899 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.896+0000 7f58bd9b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:43.900 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:43.896+0000 7f58bd9b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:51:44.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:51:44.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:51:44.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:44.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:44.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:51:44.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:44.312+0000 7f58bd9b0a40 -1 Falling back to public interface 2026-02-20T16:51:44.895 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:51:44.896+0000 7f58bd9b0a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:51:45.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:51:45.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:51:45.192 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:51:45.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:51:45.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:51:45.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:51:45.347 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1718574362,v1:127.0.0.1:6811/1718574362] [v2:127.0.0.1:6812/1718574362,v1:127.0.0.1:6813/1718574362] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:51:45.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:51:45.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:51:45.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:51:45.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6447: TEST_dual_store_replicated_cluster: create_rbd_pool 2026-02-20T16:51:45.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:51:45.492 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:51:45.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:51:45.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:51:45.721 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:51:45.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:46.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:51:47.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6448: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-20T16:51:47.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:47.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:47.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:47.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:47.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:47.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:47.300 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:47.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:47.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:47.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:47.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-20T16:51:47.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-20T16:51:47.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-20T16:51:47.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:47.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:47.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:51:47.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:51:47.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705667' 2026-02-20T16:51:47.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:47.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-20T16:51:47.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:47.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:47.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-20T16:51:47.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:47.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-20T16:51:47.451 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836486 2026-02-20T16:51:47.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-20T16:51:47.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:47.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-20T16:51:47.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:47.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:51:47.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:47.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:47.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:51:47.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:47.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:51:47.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:51:47.611 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:51:47.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:47.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:51:47.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:47.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:47.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:47.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:51:47.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:47.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:48.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:51:48.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:48.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:48.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:48.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:51:48.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:48.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:48.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6450: TEST_dual_store_replicated_cluster: create_pool foo 1 2026-02-20T16:51:48.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-20T16:51:48.517 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T16:51:48.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:49.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6451: TEST_dual_store_replicated_cluster: create_pool csr_pool 1 1 2026-02-20T16:51:49.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-20T16:51:49.740 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' created 2026-02-20T16:51:49.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:51:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6452: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-20T16:51:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:51:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:51:50.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:51:50.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:51:50.755 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:51:50.756 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:51:50.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:51:50.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:51:50.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:51:50.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:51:51.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:51:51.034 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:51:51.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:51:51.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:51.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:51:51.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:51:51.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:51:51.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:51:51.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:51:51.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:51:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-20T16:51:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-20T16:51:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705672' 2026-02-20T16:51:51.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:51.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:51:51.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:51.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:51:51.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:51:51.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:51.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:51:51.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:51:51.185 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T16:51:51.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:51.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836491 2026-02-20T16:51:51.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:51:52.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:51:52.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:51:52.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:51:52.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:51:52.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-20T16:51:52.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:51:52.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:51:52.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-20T16:51:52.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:51:52.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-20T16:51:52.524 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705672 2026-02-20T16:51:52.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-20T16:51:52.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:51:52.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-20T16:51:52.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:51:52.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:52.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:51:52.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:51:52.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:51:52.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:51:53.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:51:53.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:51:53.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:51:53.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:51:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:51:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:51:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:51:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6454: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool noscrub 1 2026-02-20T16:51:53.443 INFO:tasks.workunit.client.0.trial013.stderr:set pool 3 noscrub to 1 2026-02-20T16:51:53.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6455: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool nodeep-scrub 1 2026-02-20T16:51:53.649 INFO:tasks.workunit.client.0.trial013.stderr:set pool 3 nodeep-scrub to 1 2026-02-20T16:51:53.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-20T16:51:53.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:53.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-20T16:51:53.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:53.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:53.852 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:53.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:54.053 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:54.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-20T16:51:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-20T16:51:54.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:54.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:54.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:54.344 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:54.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:54.547 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:54.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:54.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:54.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:54.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-20T16:51:54.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-20T16:51:54.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:54.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:54.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:54.842 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:54.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:55.046 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:55.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:55.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:55.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:55.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-20T16:51:55.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-20T16:51:55.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:55.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-20T16:51:55.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:55.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:55.339 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:55.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:55.543 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:55.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:55.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:55.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:55.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-20T16:51:55.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:55.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:55.831 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:55.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:56.036 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:56.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:56.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:56.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:56.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-20T16:51:56.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:56.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-20T16:51:56.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:56.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:56.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:56.331 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:56.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:56.535 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:56.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:56.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:56.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:56.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-20T16:51:56.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-20T16:51:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-20T16:51:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-20T16:51:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:56.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-20T16:51:56.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:56.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:56.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:56.826 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:56.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:57.031 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:57.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:57.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:57.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:57.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-20T16:51:57.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-20T16:51:57.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:57.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:57.310 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:57.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:57.515 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:57.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:57.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:57.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:57.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-20T16:51:57.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:57.808 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:57.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:58.014 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:58.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:58.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:58.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:58.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-20T16:51:58.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:58.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:58.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:58.309 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:58.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:58.515 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:58.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:58.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:58.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:58.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-20T16:51:58.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-20T16:51:58.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:58.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-20T16:51:58.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:58.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:58.815 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:58.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:59.021 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:59.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:59.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:59.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:59.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-20T16:51:59.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:59.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:59.318 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:59.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:51:59.523 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:51:59.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:51:59.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:51:59.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:51:59.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-20T16:51:59.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-20T16:51:59.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:51:59.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:51:59.823 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:51:59.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:00.028 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:00.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:00.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:00.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:00.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-20T16:52:00.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-20T16:52:00.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:00.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:00.329 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:00.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:00.534 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:00.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:00.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:00.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:00.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-20T16:52:00.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-20T16:52:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:00.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:00.823 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:00.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:01.028 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:01.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:01.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:01.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:01.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-20T16:52:01.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-20T16:52:01.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:01.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:01.327 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:01.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:01.532 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:01.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:01.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:01.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:01.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-20T16:52:01.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:01.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-20T16:52:01.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:01.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:01.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:01.833 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:01.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:02.038 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:02.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:02.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:02.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:02.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-20T16:52:02.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:02.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:02.335 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:02.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:02.537 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:02.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:02.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:02.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:02.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-20T16:52:02.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-20T16:52:02.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:52:02.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:52:02.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:52:02.840 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:52:02.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:52:03.046 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:52:03.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:52:03.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:52:03.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:52:03.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-20T16:52:03.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-20T16:52:03.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6466: TEST_dual_store_replicated_cluster: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-20T16:52:03.140 INFO:tasks.workunit.client.0.trial013.stderr:1025+0 records in 2026-02-20T16:52:03.140 INFO:tasks.workunit.client.0.trial013.stderr:1025+0 records out 2026-02-20T16:52:03.140 INFO:tasks.workunit.client.0.trial013.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00405815 s, 259 MB/s 2026-02-20T16:52:03.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6467: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-20T16:52:03.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6468: TEST_dual_store_replicated_cluster: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-20T16:52:03.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-20T16:52:03.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-20T16:52:03.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-20T16:52:03.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:52:03.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:52:03.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-20T16:52:03.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: get_primary csr_pool ROBJ0 2026-02-20T16:52:03.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-20T16:52:03.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-20T16:52:03.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:52:03.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:52:03.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: local primary=1 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: get_asok_path osd.0 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:03.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:52:03.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-20T16:52:03.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-20T16:52:03.561 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:52:03.562 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-20T16:52:03.562 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:52:03.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: get_asok_path osd.1 2026-02-20T16:52:03.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:52:03.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:52:03.569 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:52:03.569 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:03.569 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:03.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:52:03.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-20T16:52:03.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-20T16:52:03.624 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:52:03.625 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-20T16:52:03.625 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:52:03.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6478: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-20T16:52:03.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:52:03.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:52:03.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:52:03.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:52:03.632 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:03.632 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:03.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:03.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:03.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:03.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:04.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:04.074 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:04.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:04.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:04.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:04.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-20T16:52:04.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-20T16:52:04.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-20T16:52:04.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:04.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:04.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705686 2026-02-20T16:52:04.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705686 2026-02-20T16:52:04.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705686' 2026-02-20T16:52:04.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:04.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-20T16:52:04.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:04.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:04.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-20T16:52:04.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:04.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-20T16:52:04.230 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836505 2026-02-20T16:52:04.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-20T16:52:04.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:04.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-20T16:52:04.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:05.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:05.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:05.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-20T16:52:05.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:06.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:52:06.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:06.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836505 2026-02-20T16:52:06.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:06.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705686 2026-02-20T16:52:06.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:06.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:06.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705686 2026-02-20T16:52:06.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:06.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705686 2026-02-20T16:52:06.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705686' 2026-02-20T16:52:06.725 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705686 2026-02-20T16:52:06.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:06.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705687 -lt 38654705686 2026-02-20T16:52:06.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:52:06.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:52:06.884 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:52:06.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:52:06.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:52:06.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:52:06.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:52:06.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:52:06.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:52:06.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:52:06.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:52:06.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:52:06.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:52:06.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:52:06.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:52:06.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:52:06.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:52:07.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:51:49.741479+0000 2026-02-20T16:52:07.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:52:07.268 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:51:49.741479+0000 last_deep_scrub_stamp 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:51:49.741479+0000 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:52:07.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:52:07.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:52:07.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:52:07.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:52:07.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:52:07.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:52:07.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:49.741479+0000 '>' 2026-02-20T16:51:49.741479+0000 2026-02-20T16:52:07.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:52:08.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:52:08.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:52:08.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:52:08.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:52:08.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:52:08.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:52:08.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:52:08.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:51:49.741479+0000 '>' 2026-02-20T16:51:49.741479+0000 2026-02-20T16:52:08.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:52:09.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:52:09.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:52:09.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:52:09.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:52:09.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:52:09.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:52:09.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:52:09.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:51:49.741479+0000 2026-02-20T16:52:09.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:52:09.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-20T16:52:09.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:09.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-20T16:52:09.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 1 % 2 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6489: TEST_dual_store_replicated_cluster: local payload=UVWXYZZZ 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6490: TEST_dual_store_replicated_cluster: echo UVWXYZZZ 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6491: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:09.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:09.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:09.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:09.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:09.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:09.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:10.265 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#-5:00000000:::deep_scrub_3.0:head#, (61) No data available 2026-02-20T16:52:10.265 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2026-02-20T16:52:10.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:10.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:10.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:10.365 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:10.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:10.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:10.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:10.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:10.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:10.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:10.389 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:10.387+0000 7fc864405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:10.391 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:10.391+0000 7fc864405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:10.392 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:10.391+0000 7fc864405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:10.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:10.779 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:10.779+0000 7fc864405a40 -1 Falling back to public interface 2026-02-20T16:52:11.354 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:11.351+0000 7fc864405a40 -1 osd.1 63 log_to_monitors true 2026-02-20T16:52:11.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:11.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:11.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:11.694 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:11.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:11.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [9,63) [v2:127.0.0.1:6810/193860127,v1:127.0.0.1:6811/193860127] [v2:127.0.0.1:6812/193860127,v1:127.0.0.1:6813/193860127] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:11.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:11.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:11.852 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:11.852 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:11.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:11.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:11.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:11.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:11.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:11.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:12.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:12.133 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:12.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:12.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:12.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:12.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-20T16:52:12.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-20T16:52:12.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-20T16:52:12.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:12.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:12.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-20T16:52:12.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-20T16:52:12.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-287762808834' 2026-02-20T16:52:12.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:12.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-20T16:52:12.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:12.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:12.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-20T16:52:12.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:12.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-20T16:52:12.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-20T16:52:12.283 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836514 2026-02-20T16:52:12.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836514 2026-02-20T16:52:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:13.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:13.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:13.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836514 2026-02-20T16:52:13.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:13.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-20T16:52:13.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:13.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:13.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-20T16:52:13.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:13.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-20T16:52:13.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-20T16:52:13.610 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 287762808834 2026-02-20T16:52:13.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:13.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808836 -lt 287762808834 2026-02-20T16:52:13.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:13.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:13.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:13.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:13.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:13.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:13.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:13.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:13.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:13.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:13.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:14.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:14.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:14.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:14.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:14.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:14.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:14.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:14.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:14.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-20T16:52:14.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 2 % 2 2026-02-20T16:52:14.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:52:14.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:14.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6496: TEST_dual_store_replicated_cluster: local payload=UVWXYZ 2026-02-20T16:52:14.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6497: TEST_dual_store_replicated_cluster: echo UVWXYZ 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6498: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:14.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:52:14.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:14.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:14.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:14.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:14.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:14.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:14.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:14.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:52:14.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:52:14.941 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:52:14.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:52:14.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:52:14.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:14.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:14.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:14.959 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:14.959+0000 7fb10cb91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:14.960 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:14.959+0000 7fb10cb91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:14.961 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:14.963+0000 7fb10cb91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:15.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:52:15.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:15.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:15.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:15.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:15.343 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:15.343+0000 7fb10cb91a40 -1 Falling back to public interface 2026-02-20T16:52:15.980 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:15.979+0000 7fb10cb91a40 -1 osd.0 68 log_to_monitors true 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6800/2205605633,v1:127.0.0.1:6801/2205605633] [v2:127.0.0.1:6802/2205605633,v1:127.0.0.1:6803/2205605633] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:16.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:16.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:16.420 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:16.420 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:16.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:16.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:16.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:16.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:16.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:16.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:16.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:16.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:16.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:16.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:16.702 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:16.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:16.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:16.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-20T16:52:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-20T16:52:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314' 2026-02-20T16:52:16.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:16.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:16.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808840 2026-02-20T16:52:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808840 2026-02-20T16:52:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314 1-287762808840' 2026-02-20T16:52:16.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:16.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645314 2026-02-20T16:52:16.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:16.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:16.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645314 2026-02-20T16:52:16.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:16.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-20T16:52:16.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2026-02-20T16:52:16.858 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 309237645314 2026-02-20T16:52:16.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:17.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 309237645314 2026-02-20T16:52:17.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:18.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:18.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:18.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645315 -lt 309237645314 2026-02-20T16:52:18.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:18.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808840 2026-02-20T16:52:18.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:18.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:18.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808840 2026-02-20T16:52:18.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:18.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808840 2026-02-20T16:52:18.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808840' 2026-02-20T16:52:18.185 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 287762808840 2026-02-20T16:52:18.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:18.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808841 -lt 287762808840 2026-02-20T16:52:18.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:18.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:18.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:18.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:18.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:18.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:18.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:18.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:18.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:18.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:18.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:18.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:18.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:18.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-20T16:52:18.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 3 % 2 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6503: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:52:18.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:18.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:19.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:19.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-20T16:52:19.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:19.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:19.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:19.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:19.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:19.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-20T16:52:19.402 INFO:tasks.workunit.client.0.trial013.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-20T16:52:19.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:19.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:19.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:19.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:19.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:19.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:19.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:19.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:19.501 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:19.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:19.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:19.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:19.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:19.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:19.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:19.517 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:19.515+0000 7f14a6e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:19.518 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:19.519+0000 7f14a6e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:19.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:19.519+0000 7f14a6e64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:19.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:19.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:19.943 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:19.943+0000 7f14a6e64a40 -1 Falling back to public interface 2026-02-20T16:52:20.518 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:20.515+0000 7f14a6e64a40 -1 osd.1 74 log_to_monitors true 2026-02-20T16:52:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:20.816 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:20.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:20.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:20.974 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [67,74) [v2:127.0.0.1:6810/1959327254,v1:127.0.0.1:6811/1959327254] [v2:127.0.0.1:6812/1959327254,v1:127.0.0.1:6813/1959327254] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:20.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:20.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:20.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:20.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:20.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:20.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:20.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:20.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:20.976 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:20.976 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:20.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:20.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:20.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:21.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:21.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:21.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:21.253 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645320 2026-02-20T16:52:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645320 2026-02-20T16:52:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645320' 2026-02-20T16:52:21.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:21.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:21.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-20T16:52:21.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-20T16:52:21.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645320 1-335007449090' 2026-02-20T16:52:21.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:21.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645320 2026-02-20T16:52:21.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:21.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:21.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645320 2026-02-20T16:52:21.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:21.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645320 2026-02-20T16:52:21.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645320' 2026-02-20T16:52:21.405 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 309237645320 2026-02-20T16:52:21.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:21.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645317 -lt 309237645320 2026-02-20T16:52:21.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:22.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:22.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:22.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645320 -lt 309237645320 2026-02-20T16:52:22.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:22.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449090 2026-02-20T16:52:22.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:22.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:22.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449090 2026-02-20T16:52:22.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:22.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-20T16:52:22.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2026-02-20T16:52:22.739 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 335007449090 2026-02-20T16:52:22.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:22.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449090 -lt 335007449090 2026-02-20T16:52:22.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:22.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:22.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:23.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:23.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:23.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:23.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:23.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:23.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:23.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-20T16:52:23.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 4 % 2 2026-02-20T16:52:23.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6508: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:52:23.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:23.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:23.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:23.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:24.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:24.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:24.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:24.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:24.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:24.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:24.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:52:24.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:52:24.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:24.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:24.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:24.060 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:24.059+0000 7f8a14f2fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:24.061 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:24.059+0000 7f8a14f2fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:24.062 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:24.063+0000 7f8a14f2fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:24.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:52:24.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:24.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:52:24.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:24.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:24.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:24.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:24.519+0000 7f8a14f2fa40 -1 Falling back to public interface 2026-02-20T16:52:25.094 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:25.091+0000 7f8a14f2fa40 -1 osd.0 80 log_to_monitors true 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:25.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:25.524 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [72,80) [v2:127.0.0.1:6800/1174937132,v1:127.0.0.1:6801/1174937132] [v2:127.0.0.1:6802/1174937132,v1:127.0.0.1:6803/1174937132] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:52:25.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:25.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:25.526 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:25.526 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:25.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:25.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:25.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:25.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:25.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:25.806 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:25.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:25.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:25.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:25.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-20T16:52:25.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-20T16:52:25.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866' 2026-02-20T16:52:25.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:25.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449096 2026-02-20T16:52:25.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449096 2026-02-20T16:52:25.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-335007449096' 2026-02-20T16:52:25.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:25.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252866 2026-02-20T16:52:25.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:25.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:25.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252866 2026-02-20T16:52:25.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:25.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-20T16:52:25.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2026-02-20T16:52:25.957 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 360777252866 2026-02-20T16:52:25.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:26.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 360777252866 2026-02-20T16:52:26.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:27.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:27.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:27.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 360777252866 2026-02-20T16:52:27.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:28.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:52:28.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:28.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252868 -lt 360777252866 2026-02-20T16:52:28.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:28.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449096 2026-02-20T16:52:28.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:28.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:28.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449096 2026-02-20T16:52:28.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:28.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449096 2026-02-20T16:52:28.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449096' 2026-02-20T16:52:28.449 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 335007449096 2026-02-20T16:52:28.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:28.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449097 -lt 335007449096 2026-02-20T16:52:28.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:28.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:28.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:28.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:28.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:28.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:28.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:28.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:28.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:28.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:28.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:28.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:28.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:28.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:29.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:29.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:29.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:29.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:29.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-20T16:52:29.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 5 % 2 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6513: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:29.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:29.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:29.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:52:29.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:29.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:29.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:29.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:29.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:29.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:29.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:29.761 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:29.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:29.762 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:29.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:29.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:29.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:29.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:29.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:29.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:29.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:29.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:29.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:29.783 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:29.783+0000 7f2123c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:29.785 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:29.783+0000 7f2123c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:29.786 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:29.787+0000 7f2123c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:29.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:29.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:29.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:30.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:30.231 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:30.231+0000 7f2123c51a40 -1 Falling back to public interface 2026-02-20T16:52:30.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:30.919+0000 7f2123c51a40 -1 osd.1 86 log_to_monitors true 2026-02-20T16:52:31.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:31.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:31.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:31.086 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:31.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:31.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:31.242 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6810/3770651982,v1:127.0.0.1:6811/3770651982] [v2:127.0.0.1:6812/3770651982,v1:127.0.0.1:6813/3770651982] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:31.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:31.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:31.244 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:31.244 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:31.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:31.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:31.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:31.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:31.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:31.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:31.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:31.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:31.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:31.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:31.532 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:31.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:31.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:31.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252873 2026-02-20T16:52:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252873 2026-02-20T16:52:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252873' 2026-02-20T16:52:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:31.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:31.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-20T16:52:31.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-20T16:52:31.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252873 1-386547056642' 2026-02-20T16:52:31.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:31.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252873 2026-02-20T16:52:31.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:31.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:31.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252873 2026-02-20T16:52:31.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:31.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252873 2026-02-20T16:52:31.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252873' 2026-02-20T16:52:31.683 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 360777252873 2026-02-20T16:52:31.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:31.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252872 -lt 360777252873 2026-02-20T16:52:31.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:32.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:32.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:33.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252872 -lt 360777252873 2026-02-20T16:52:33.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:34.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:52:34.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:34.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252875 -lt 360777252873 2026-02-20T16:52:34.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:34.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056642 2026-02-20T16:52:34.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:34.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:34.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056642 2026-02-20T16:52:34.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:34.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-20T16:52:34.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2026-02-20T16:52:34.164 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 386547056642 2026-02-20T16:52:34.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:34.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056644 -lt 386547056642 2026-02-20T16:52:34.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:34.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:34.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:34.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:34.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:34.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:34.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:34.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:34.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:34.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-20T16:52:34.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 6 % 2 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6518: TEST_dual_store_replicated_cluster: echo extra 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6519: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:52:34.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:34.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:34.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:34.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:52:35.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:35.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:35.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:52:35.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:35.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:35.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-20T16:52:35.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:35.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:35.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:35.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:52:35.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:52:35.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:35.507 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:35.506+0000 7f4012837a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:35.508 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:35.506+0000 7f4012837a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:35.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:35.506+0000 7f4012837a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:35.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:52:35.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:35.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:35.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:35.871 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:35.870+0000 7f4012837a40 -1 Falling back to public interface 2026-02-20T16:52:36.534 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:36.534+0000 7f4012837a40 -1 osd.0 91 log_to_monitors true 2026-02-20T16:52:36.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:36.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:36.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:36.810 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:36.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:36.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:36.970 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6800/1458122204,v1:127.0.0.1:6801/1458122204] [v2:127.0.0.1:6802/1458122204,v1:127.0.0.1:6803/1458122204] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:36.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:36.972 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:36.972 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:36.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:36.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:36.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:37.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:37.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:37.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:37.244 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:37.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:37.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:37.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:37.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-20T16:52:37.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-20T16:52:37.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122' 2026-02-20T16:52:37.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:37.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:37.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056649 2026-02-20T16:52:37.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056649 2026-02-20T16:52:37.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122 1-386547056649' 2026-02-20T16:52:37.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:37.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-408021893122 2026-02-20T16:52:37.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:37.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-408021893122 2026-02-20T16:52:37.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-20T16:52:37.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2026-02-20T16:52:37.397 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 408021893122 2026-02-20T16:52:37.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:37.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252876 -lt 408021893122 2026-02-20T16:52:37.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:38.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:38.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:38.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893122 -lt 408021893122 2026-02-20T16:52:38.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:38.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056649 2026-02-20T16:52:38.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:38.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:38.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056649 2026-02-20T16:52:38.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:38.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056649 2026-02-20T16:52:38.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056649' 2026-02-20T16:52:38.731 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 386547056649 2026-02-20T16:52:38.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:38.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056649 -lt 386547056649 2026-02-20T16:52:38.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:38.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:38.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:39.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:39.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:39.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:39.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:39.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:39.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:39.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:39.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:39.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:39.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:39.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6520: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/extra-val 2026-02-20T16:52:39.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:39.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-20T16:52:39.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 7 % 2 2026-02-20T16:52:39.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6525: TEST_dual_store_replicated_cluster: echo -n newheader 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6526: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:39.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:39.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:39.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:39.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:39.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-20T16:52:40.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:40.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:40.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:40.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:40.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:40.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:40.042 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:40.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:40.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:40.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:40.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:40.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:40.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:40.058 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:40.058+0000 7f214efd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:40.059 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:40.058+0000 7f214efd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:40.061 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:40.058+0000 7f214efd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:40.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:40.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:40.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:40.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:40.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:40.527 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:40.526+0000 7f214efd4a40 -1 Falling back to public interface 2026-02-20T16:52:41.153 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:41.150+0000 7f214efd4a40 -1 osd.1 97 log_to_monitors true 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:41.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:41.521 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/27226553,v1:127.0.0.1:6811/27226553] [v2:127.0.0.1:6812/27226553,v1:127.0.0.1:6813/27226553] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:41.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:41.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:41.523 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:41.523 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:41.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:41.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:41.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:41.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:41.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:41.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:41.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:41.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:41.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:41.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:41.799 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:41.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:41.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:41.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:41.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893128 2026-02-20T16:52:41.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893128 2026-02-20T16:52:41.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893128' 2026-02-20T16:52:41.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:41.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:41.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-20T16:52:41.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-20T16:52:41.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893128 1-433791696898' 2026-02-20T16:52:41.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:41.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-408021893128 2026-02-20T16:52:41.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:41.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:41.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-408021893128 2026-02-20T16:52:41.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:41.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893128 2026-02-20T16:52:41.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 408021893128' 2026-02-20T16:52:41.952 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 408021893128 2026-02-20T16:52:41.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:42.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893126 -lt 408021893128 2026-02-20T16:52:42.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:43.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:43.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:43.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893126 -lt 408021893128 2026-02-20T16:52:43.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:44.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:52:44.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:44.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893129 -lt 408021893128 2026-02-20T16:52:44.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:44.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-433791696898 2026-02-20T16:52:44.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:44.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:44.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-433791696898 2026-02-20T16:52:44.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:44.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-20T16:52:44.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2026-02-20T16:52:44.455 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 433791696898 2026-02-20T16:52:44.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:44.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696900 -lt 433791696898 2026-02-20T16:52:44.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:44.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:44.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:44.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:44.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:44.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:44.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:44.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:44.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:44.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:44.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:44.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:44.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:44.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:44.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:45.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:45.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:45.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:45.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6527: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/hdr 2026-02-20T16:52:45.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:45.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-20T16:52:45.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 8 % 2 2026-02-20T16:52:45.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:52:45.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:45.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6531: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-20T16:52:45.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6532: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6535: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6536: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:52:45.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:45.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:45.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:45.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:45.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:45.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-20T16:52:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:45.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:45.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:45.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:52:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:52:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:45.839 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:45.838+0000 7f895b206a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:45.840 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:45.838+0000 7f895b206a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:45.841 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:45.838+0000 7f895b206a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:45.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:45.983 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:45.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:45.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:46.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:46.235 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:46.234+0000 7f895b206a40 -1 Falling back to public interface 2026-02-20T16:52:46.900 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:46.898+0000 7f895b206a40 -1 osd.0 102 log_to_monitors true 2026-02-20T16:52:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:47.140 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:47.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6800/3898688457,v1:127.0.0.1:6801/3898688457] [v2:127.0.0.1:6802/3898688457,v1:127.0.0.1:6803/3898688457] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:47.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:47.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:47.296 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:47.296 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:47.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:47.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:47.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:47.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:47.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:47.571 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:47.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:47.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:47.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:47.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-20T16:52:47.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-20T16:52:47.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533378' 2026-02-20T16:52:47.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:47.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:47.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696905 2026-02-20T16:52:47.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696905 2026-02-20T16:52:47.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533378 1-433791696905' 2026-02-20T16:52:47.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:47.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533378 2026-02-20T16:52:47.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:47.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:47.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533378 2026-02-20T16:52:47.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:47.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-20T16:52:47.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2026-02-20T16:52:47.726 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 455266533378 2026-02-20T16:52:47.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:47.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 455266533378 2026-02-20T16:52:47.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:48.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:48.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:49.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 455266533378 2026-02-20T16:52:49.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:50.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:52:50.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:50.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533380 -lt 455266533378 2026-02-20T16:52:50.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:50.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-433791696905 2026-02-20T16:52:50.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:50.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:50.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-433791696905 2026-02-20T16:52:50.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:50.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696905 2026-02-20T16:52:50.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 433791696905' 2026-02-20T16:52:50.210 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 433791696905 2026-02-20T16:52:50.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:50.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696907 -lt 433791696905 2026-02-20T16:52:50.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:50.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:50.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:50.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:50.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:50.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:50.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:50.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:50.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:50.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:50.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:50.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:50.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:50.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:50.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:50.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:50.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:50.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:50.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6537: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:52:50.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:50.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:50.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:50.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:50.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:50.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:50.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:51.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:51.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:51.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:51.590 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:51.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:51.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:51.608 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:51.606+0000 7f1353765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:51.609 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:51.606+0000 7f1353765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:51.610 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:51.610+0000 7f1353765a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:51.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:51.755 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:51.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:51.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:51.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:52.051 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:52.050+0000 7f1353765a40 -1 Falling back to public interface 2026-02-20T16:52:52.730 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:52.730+0000 7f1353765a40 -1 osd.1 108 log_to_monitors true 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:52.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:53.071 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/1278177415,v1:127.0.0.1:6811/1278177415] [v2:127.0.0.1:6812/1278177415,v1:127.0.0.1:6813/1278177415] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:53.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:53.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:53.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:53.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:53.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:53.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:53.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:53.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:53.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:53.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:53.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:53.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:53.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:53.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:53.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:53.357 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:53.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:53.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:53.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:53.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533385 2026-02-20T16:52:53.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533385 2026-02-20T16:52:53.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533385' 2026-02-20T16:52:53.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:53.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=481036337154 2026-02-20T16:52:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 481036337154 2026-02-20T16:52:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533385 1-481036337154' 2026-02-20T16:52:53.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:53.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533385 2026-02-20T16:52:53.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:53.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:53.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533385 2026-02-20T16:52:53.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:53.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533385 2026-02-20T16:52:53.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533385' 2026-02-20T16:52:53.510 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 455266533385 2026-02-20T16:52:53.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:53.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533385 -lt 455266533385 2026-02-20T16:52:53.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:53.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-481036337154 2026-02-20T16:52:53.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:53.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:53.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-481036337154 2026-02-20T16:52:53.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:53.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=481036337154 2026-02-20T16:52:53.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2026-02-20T16:52:53.690 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 481036337154 2026-02-20T16:52:53.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:53.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 481036337154 -lt 481036337154 2026-02-20T16:52:53.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:53.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:53.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:54.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:54.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:54.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:54.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:54.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:54.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:54.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:54.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6538: TEST_dual_store_replicated_cluster: echo -n val3-ROBJ8 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6539: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:54.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:54.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:54.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:52:54.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:54.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:54.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:54.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:54.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:54.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-20T16:52:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:54.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:54.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:54.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:54.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:54.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:54.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:55.017 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:55.014+0000 7f728ac14a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:55.018 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:55.018+0000 7f728ac14a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:55.019 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:55.018+0000 7f728ac14a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:55.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:55.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:55.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:55.408 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:55.406+0000 7f728ac14a40 -1 Falling back to public interface 2026-02-20T16:52:56.024 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:56.022+0000 7f728ac14a40 -1 osd.1 113 log_to_monitors true 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/2168405578,v1:127.0.0.1:6811/2168405578] [v2:127.0.0.1:6812/2168405578,v1:127.0.0.1:6813/2168405578] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:52:56.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:52:56.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:52:56.476 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:52:56.477 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:52:56.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:52:56.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:52:56.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:52:56.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:52:56.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:52:56.757 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:52:56.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:52:56.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:56.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:52:56.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533389 2026-02-20T16:52:56.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533389 2026-02-20T16:52:56.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533389' 2026-02-20T16:52:56.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:52:56.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:52:56.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=502511173634 2026-02-20T16:52:56.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 502511173634 2026-02-20T16:52:56.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533389 1-502511173634' 2026-02-20T16:52:56.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:56.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533389 2026-02-20T16:52:56.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:56.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:52:56.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533389 2026-02-20T16:52:56.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:56.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533389 2026-02-20T16:52:56.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533389' 2026-02-20T16:52:56.907 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 455266533389 2026-02-20T16:52:56.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:57.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533387 -lt 455266533389 2026-02-20T16:52:57.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:52:58.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:52:58.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:52:58.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533390 -lt 455266533389 2026-02-20T16:52:58.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:52:58.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-502511173634 2026-02-20T16:52:58.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:52:58.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:52:58.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-502511173634 2026-02-20T16:52:58.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:52:58.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=502511173634 2026-02-20T16:52:58.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2026-02-20T16:52:58.229 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 502511173634 2026-02-20T16:52:58.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:52:58.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 502511173635 -lt 502511173634 2026-02-20T16:52:58.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:52:58.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:58.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:58.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:52:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:52:58.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:52:58.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:52:58.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:52:58.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:52:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:52:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:52:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:52:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6540: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-20T16:52:58.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:52:58.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-20T16:52:58.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 9 % 2 2026-02-20T16:52:58.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:52:58.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:52:58.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6544: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:52:58.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:52:58.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:52:58.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:52:58.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:59.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:52:59.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:52:59.438 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:52:59.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:52:59.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:52:59.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:52:59.458 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:59.454+0000 7f03a8c0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:59.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:59.458+0000 7f03a8c0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:59.460 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:59.458+0000 7f03a8c0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:52:59.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:52:59.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:52:59.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:52:59.892 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:52:59.890+0000 7f03a8c0fa40 -1 Falling back to public interface 2026-02-20T16:53:00.582 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:00.582+0000 7f03a8c0fa40 -1 osd.1 119 log_to_monitors true 2026-02-20T16:53:00.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:00.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:00.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:00.764 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:00.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:00.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:00.923 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 123 up_thru 123 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/2305003977,v1:127.0.0.1:6811/2305003977] [v2:127.0.0.1:6812/2305003977,v1:127.0.0.1:6813/2305003977] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:00.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:00.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:00.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:00.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:00.925 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:00.925 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:01.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:01.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:01.199 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:01.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:01.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:01.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533395 2026-02-20T16:53:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533395 2026-02-20T16:53:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533395' 2026-02-20T16:53:01.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:01.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:01.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=528280977410 2026-02-20T16:53:01.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 528280977410 2026-02-20T16:53:01.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533395 1-528280977410' 2026-02-20T16:53:01.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:01.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533395 2026-02-20T16:53:01.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:01.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:01.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533395 2026-02-20T16:53:01.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:01.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533395 2026-02-20T16:53:01.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533395' 2026-02-20T16:53:01.350 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 455266533395 2026-02-20T16:53:01.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:01.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533392 -lt 455266533395 2026-02-20T16:53:01.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:02.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:02.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:02.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533395 -lt 455266533395 2026-02-20T16:53:02.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:02.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-528280977410 2026-02-20T16:53:02.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:02.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:02.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-528280977410 2026-02-20T16:53:02.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:02.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=528280977410 2026-02-20T16:53:02.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2026-02-20T16:53:02.679 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 528280977410 2026-02-20T16:53:02.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:02.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 528280977410 -lt 528280977410 2026-02-20T16:53:02.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:02.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:02.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:03.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:03.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:03.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:03.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:03.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:03.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:03.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:03.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:03.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:03.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:03.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6545: TEST_dual_store_replicated_cluster: echo -n D 2026-02-20T16:53:03.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6546: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-20T16:53:03.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6547: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:53:03.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:03.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:03.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:03.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:03.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-20T16:53:04.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:04.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:04.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:04.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:04.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:04.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:04.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:04.005 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:04.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:04.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:04.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:04.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:04.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:04.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:04.023 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:04.022+0000 7f428f448a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:04.025 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:04.022+0000 7f428f448a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:04.026 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:04.026+0000 7f428f448a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:04.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:04.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:04.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:04.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:04.451 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:04.450+0000 7f428f448a40 -1 Falling back to public interface 2026-02-20T16:53:05.084 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:05.081+0000 7f428f448a40 -1 osd.1 124 log_to_monitors true 2026-02-20T16:53:05.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:05.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:05.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:05.328 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:05.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:05.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 128 up_thru 128 down_at 125 last_clean_interval [123,124) [v2:127.0.0.1:6810/3751571006,v1:127.0.0.1:6811/3751571006] [v2:127.0.0.1:6812/3751571006,v1:127.0.0.1:6813/3751571006] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:05.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:05.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:05.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:05.492 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:05.492 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:05.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:05.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:05.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:05.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:05.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:05.773 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:05.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:05.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:05.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:05.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533400 2026-02-20T16:53:05.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533400 2026-02-20T16:53:05.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533400' 2026-02-20T16:53:05.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:05.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:05.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=549755813890 2026-02-20T16:53:05.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 549755813890 2026-02-20T16:53:05.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533400 1-549755813890' 2026-02-20T16:53:05.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:05.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533400 2026-02-20T16:53:05.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:05.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:05.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533400 2026-02-20T16:53:05.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:05.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533400 2026-02-20T16:53:05.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533400' 2026-02-20T16:53:05.923 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 455266533400 2026-02-20T16:53:05.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:06.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533399 -lt 455266533400 2026-02-20T16:53:06.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:07.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:07.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533399 -lt 455266533400 2026-02-20T16:53:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:08.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:08.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:08.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533402 -lt 455266533400 2026-02-20T16:53:08.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:08.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-549755813890 2026-02-20T16:53:08.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:08.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:08.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-549755813890 2026-02-20T16:53:08.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:08.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=549755813890 2026-02-20T16:53:08.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2026-02-20T16:53:08.411 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 549755813890 2026-02-20T16:53:08.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:08.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 549755813892 -lt 549755813890 2026-02-20T16:53:08.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:08.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:08.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:08.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:08.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:08.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:08.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:08.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:08.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:08.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:08.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:08.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:08.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:08.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:08.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:09.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:09.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:09.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:09.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6548: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-20T16:53:09.103 INFO:tasks.workunit.client.0.trial013.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-20T16:53:09.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:09.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-20T16:53:09.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 10 % 2 2026-02-20T16:53:09.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:53:09.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:09.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:09.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-20T16:53:09.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 11 % 2 2026-02-20T16:53:09.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:53:09.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:09.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:09.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-20T16:53:09.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 12 % 2 2026-02-20T16:53:09.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:53:09.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:09.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:09.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-20T16:53:09.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 13 % 2 2026-02-20T16:53:09.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:53:09.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:09.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:09.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-20T16:53:09.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 14 % 2 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6557: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6558: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:53:09.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:09.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:09.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-20T16:53:09.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:09.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:53:09.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:09.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:09.739 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:09.737+0000 7f46eb83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:09.741 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:09.737+0000 7f46eb83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:09.742 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:09.741+0000 7f46eb83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:09.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:53:09.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:09.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:53:09.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:09.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:10.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:10.180 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:10.177+0000 7f46eb83ca40 -1 Falling back to public interface 2026-02-20T16:53:10.780 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:10.777+0000 7f46eb83ca40 -1 osd.0 130 log_to_monitors true 2026-02-20T16:53:11.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:11.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:11.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:11.041 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:11.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:11.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:11.203 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [106,130) [v2:127.0.0.1:6800/3809681508,v1:127.0.0.1:6801/3809681508] [v2:127.0.0.1:6802/3809681508,v1:127.0.0.1:6803/3809681508] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:53:11.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:11.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:11.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:11.205 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:11.205 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:11.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:11.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:11.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:11.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:11.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:11.484 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:11.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:11.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:11.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:11.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617666 2026-02-20T16:53:11.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617666 2026-02-20T16:53:11.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617666' 2026-02-20T16:53:11.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:11.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:11.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=549755813897 2026-02-20T16:53:11.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 549755813897 2026-02-20T16:53:11.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617666 1-549755813897' 2026-02-20T16:53:11.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:11.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617666 2026-02-20T16:53:11.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:11.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:11.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617666 2026-02-20T16:53:11.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:11.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617666 2026-02-20T16:53:11.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2026-02-20T16:53:11.637 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 575525617666 2026-02-20T16:53:11.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:11.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 575525617666 2026-02-20T16:53:11.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:12.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:12.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:12.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 575525617666 2026-02-20T16:53:12.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:13.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:13.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:14.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617668 -lt 575525617666 2026-02-20T16:53:14.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:14.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-549755813897 2026-02-20T16:53:14.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:14.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:14.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-549755813897 2026-02-20T16:53:14.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:14.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=549755813897 2026-02-20T16:53:14.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 549755813897' 2026-02-20T16:53:14.136 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 549755813897 2026-02-20T16:53:14.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:14.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 549755813899 -lt 549755813897 2026-02-20T16:53:14.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:14.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:14.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:14.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:14.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:14.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:14.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:14.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:14.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:14.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:14.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:14.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:14.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:14.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:14.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:14.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:14.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:14.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6559: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:14.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:14.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:14.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:14.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:15.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:15.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-20T16:53:15.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:15.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:15.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:15.483 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:15.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:15.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:15.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:15.507 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:15.505+0000 7fd56c10ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:15.508 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:15.505+0000 7fd56c10ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:15.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:15.505+0000 7fd56c10ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:15.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:15.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:15.904 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:15.901+0000 7fd56c10ba40 -1 Falling back to public interface 2026-02-20T16:53:16.474 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:16.469+0000 7fd56c10ba40 -1 osd.1 135 log_to_monitors true 2026-02-20T16:53:16.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:16.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:16.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:16.810 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:16.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:16.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 139 up_thru 139 down_at 136 last_clean_interval [128,135) [v2:127.0.0.1:6810/3800711421,v1:127.0.0.1:6811/3800711421] [v2:127.0.0.1:6812/3800711421,v1:127.0.0.1:6813/3800711421] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:16.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:16.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:16.966 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:16.967 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:16.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:16.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:16.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:17.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:17.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:17.245 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:17.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:17.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:17.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617673 2026-02-20T16:53:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617673 2026-02-20T16:53:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617673' 2026-02-20T16:53:17.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:17.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:17.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=597000454146 2026-02-20T16:53:17.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 597000454146 2026-02-20T16:53:17.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617673 1-597000454146' 2026-02-20T16:53:17.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:17.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617673 2026-02-20T16:53:17.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:17.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:17.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617673 2026-02-20T16:53:17.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:17.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617673 2026-02-20T16:53:17.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617673' 2026-02-20T16:53:17.395 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 575525617673 2026-02-20T16:53:17.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:17.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617670 -lt 575525617673 2026-02-20T16:53:17.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:18.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:18.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:18.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617673 -lt 575525617673 2026-02-20T16:53:18.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:18.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-597000454146 2026-02-20T16:53:18.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:18.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:18.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-597000454146 2026-02-20T16:53:18.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:18.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=597000454146 2026-02-20T16:53:18.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2026-02-20T16:53:18.724 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 597000454146 2026-02-20T16:53:18.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:18.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 597000454146 -lt 597000454146 2026-02-20T16:53:18.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:18.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:18.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:19.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:19.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:19.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:19.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:19.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:19.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:19.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:19.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:19.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:19.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:19.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:19.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:19.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:19.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:19.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:19.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6560: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val 2026-02-20T16:53:19.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:19.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-20T16:53:19.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 15 % 2 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6564: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:19.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:19.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:19.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:19.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:19.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:19.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:19.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-20T16:53:19.966 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:53:20.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:20.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:20.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:20.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:20.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:20.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:20.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:20.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:20.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:20.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:20.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:20.092 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:20.089+0000 7f6475456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:20.094 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:20.089+0000 7f6475456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:20.094 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:20.093+0000 7f6475456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:20.235 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:20.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:20.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:20.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:20.476 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:20.473+0000 7f6475456a40 -1 Falling back to public interface 2026-02-20T16:53:21.173 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:21.169+0000 7f6475456a40 -1 osd.1 140 log_to_monitors true 2026-02-20T16:53:21.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:21.393 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:21.552 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [139,140) [v2:127.0.0.1:6810/4190974142,v1:127.0.0.1:6811/4190974142] [v2:127.0.0.1:6812/4190974142,v1:127.0.0.1:6813/4190974142] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:21.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:21.554 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:21.554 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:21.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:21.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:21.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:21.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:21.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:21.831 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:21.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:21.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:21.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:21.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617679 2026-02-20T16:53:21.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617679 2026-02-20T16:53:21.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617679' 2026-02-20T16:53:21.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:21.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290626 2026-02-20T16:53:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290626 2026-02-20T16:53:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617679 1-618475290626' 2026-02-20T16:53:21.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:21.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617679 2026-02-20T16:53:21.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:21.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:21.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617679 2026-02-20T16:53:21.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:21.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617679 2026-02-20T16:53:21.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617679' 2026-02-20T16:53:21.983 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 575525617679 2026-02-20T16:53:21.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:22.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617677 -lt 575525617679 2026-02-20T16:53:22.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:23.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:23.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617677 -lt 575525617679 2026-02-20T16:53:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:24.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:24.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:24.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617680 -lt 575525617679 2026-02-20T16:53:24.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:24.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290626 2026-02-20T16:53:24.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:24.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:24.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290626 2026-02-20T16:53:24.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:24.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290626 2026-02-20T16:53:24.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2026-02-20T16:53:24.473 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 618475290626 2026-02-20T16:53:24.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:24.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290628 -lt 618475290626 2026-02-20T16:53:24.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:24.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:24.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:24.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:24.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:24.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:24.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:24.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:24.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:24.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:24.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:25.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:25.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:25.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:25.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:25.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-20T16:53:25.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 16 % 2 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6568: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:25.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:25.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:25.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:25.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-20T16:53:25.675 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:53:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:25.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:25.786 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:25.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:25.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:25.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:25.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:25.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:53:25.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:25.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:53:25.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:53:25.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:25.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:25.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:25.807 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:25.805+0000 7f42f858ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:25.809 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:25.805+0000 7f42f858ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:25.810 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:25.805+0000 7f42f858ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:25.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:25.948 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:25.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:25.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:26.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:26.160 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:26.157+0000 7f42f858ba40 -1 Falling back to public interface 2026-02-20T16:53:26.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:26.745+0000 7f42f858ba40 -1 osd.0 145 log_to_monitors true 2026-02-20T16:53:27.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:27.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:27.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:27.114 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:27.274 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [134,145) [v2:127.0.0.1:6800/823345218,v1:127.0.0.1:6801/823345218] [v2:127.0.0.1:6802/823345218,v1:127.0.0.1:6803/823345218] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:27.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:27.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:27.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:27.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:27.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:27.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:27.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:27.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:27.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:27.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:27.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:27.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:27.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:27.548 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:27.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:27.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:27.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:27.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127106 2026-02-20T16:53:27.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127106 2026-02-20T16:53:27.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106' 2026-02-20T16:53:27.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:27.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:27.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290633 2026-02-20T16:53:27.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290633 2026-02-20T16:53:27.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106 1-618475290633' 2026-02-20T16:53:27.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:27.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127106 2026-02-20T16:53:27.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:27.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:27.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127106 2026-02-20T16:53:27.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:27.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127106 2026-02-20T16:53:27.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2026-02-20T16:53:27.703 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 639950127106 2026-02-20T16:53:27.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 639950127106 2026-02-20T16:53:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:28.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:28.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:29.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 639950127106 2026-02-20T16:53:29.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:30.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:30.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127108 -lt 639950127106 2026-02-20T16:53:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:30.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290633 2026-02-20T16:53:30.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:30.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:30.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290633 2026-02-20T16:53:30.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:30.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290633 2026-02-20T16:53:30.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290633' 2026-02-20T16:53:30.196 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 618475290633 2026-02-20T16:53:30.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:30.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290635 -lt 618475290633 2026-02-20T16:53:30.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:30.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:30.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:30.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:30.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:30.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:30.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:30.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:30.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:30.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:30.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:30.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:30.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:30.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:30.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:30.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:30.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:30.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:30.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6569: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6570: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:30.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:30.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:31.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:31.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-20T16:53:31.382 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:53:31.382 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:53:31.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:31.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:31.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:31.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:31.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:31.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:31.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:31.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:31.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:31.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:31.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:31.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:31.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:31.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:31.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:31.517+0000 7f2111b6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:31.520 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:31.517+0000 7f2111b6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:31.521 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:31.517+0000 7f2111b6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:31.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:31.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:31.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:31.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:31.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:31.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:31.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:31.666 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:31.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:31.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:31.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:31.920 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:31.917+0000 7f2111b6ba40 -1 Falling back to public interface 2026-02-20T16:53:32.539 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:32.537+0000 7f2111b6ba40 -1 osd.1 150 log_to_monitors true 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:32.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:32.989 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/3039583786,v1:127.0.0.1:6811/3039583786] [v2:127.0.0.1:6812/3039583786,v1:127.0.0.1:6813/3039583786] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:32.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:32.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:32.991 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:32.991 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:32.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:32.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:32.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:33.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:33.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:33.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:33.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:33.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:33.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:33.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:33.274 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:33.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:33.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:33.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:33.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127113 2026-02-20T16:53:33.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127113 2026-02-20T16:53:33.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127113' 2026-02-20T16:53:33.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:33.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:33.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963586 2026-02-20T16:53:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963586 2026-02-20T16:53:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127113 1-661424963586' 2026-02-20T16:53:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:33.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127113 2026-02-20T16:53:33.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:33.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:33.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127113 2026-02-20T16:53:33.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:33.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127113 2026-02-20T16:53:33.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127113' 2026-02-20T16:53:33.426 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 639950127113 2026-02-20T16:53:33.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:33.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127113 -lt 639950127113 2026-02-20T16:53:33.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:33.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963586 2026-02-20T16:53:33.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:33.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:33.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963586 2026-02-20T16:53:33.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:33.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963586 2026-02-20T16:53:33.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2026-02-20T16:53:33.605 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 661424963586 2026-02-20T16:53:33.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:33.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963586 -lt 661424963586 2026-02-20T16:53:33.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:33.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:33.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:33.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:33.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:33.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:34.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:34.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:34.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:34.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:34.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:34.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:34.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:34.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:34.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-20T16:53:34.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 17 % 2 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6575: TEST_dual_store_replicated_cluster: local payload=ROBJ17 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6576: TEST_dual_store_replicated_cluster: echo ROBJ17 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6577: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:34.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:34.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:53:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:34.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:34.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:34.817 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:53:34.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:34.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:34.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:34.952 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:34.949+0000 7f5cc5052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:34.953 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:34.949+0000 7f5cc5052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:34.954 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:34.953+0000 7f5cc5052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:35.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:53:35.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:35.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:35.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:35.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:35.384 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:35.381+0000 7f5cc5052a40 -1 Falling back to public interface 2026-02-20T16:53:35.949 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:35.945+0000 7f5cc5052a40 -1 osd.0 155 log_to_monitors true 2026-02-20T16:53:36.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:36.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:36.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:36.254 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:36.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:36.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:36.417 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 159 up_thru 159 down_at 156 last_clean_interval [149,155) [v2:127.0.0.1:6800/52653697,v1:127.0.0.1:6801/52653697] [v2:127.0.0.1:6802/52653697,v1:127.0.0.1:6803/52653697] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:53:36.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:36.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:36.419 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:36.419 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:36.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:36.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:36.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:36.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:36.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:36.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:36.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:36.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:36.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:36.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:36.696 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:36.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:36.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:36.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:36.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800066 2026-02-20T16:53:36.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800066 2026-02-20T16:53:36.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066' 2026-02-20T16:53:36.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:36.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:36.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963591 2026-02-20T16:53:36.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963591 2026-02-20T16:53:36.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066 1-661424963591' 2026-02-20T16:53:36.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:36.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800066 2026-02-20T16:53:36.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:36.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:36.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800066 2026-02-20T16:53:36.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:36.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800066 2026-02-20T16:53:36.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800066' 2026-02-20T16:53:36.846 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 682899800066 2026-02-20T16:53:36.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:37.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127114 -lt 682899800066 2026-02-20T16:53:37.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:38.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:38.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:38.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800067 -lt 682899800066 2026-02-20T16:53:38.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:38.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963591 2026-02-20T16:53:38.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:38.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:38.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963591 2026-02-20T16:53:38.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:38.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963591 2026-02-20T16:53:38.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963591' 2026-02-20T16:53:38.173 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 661424963591 2026-02-20T16:53:38.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:38.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963591 -lt 661424963591 2026-02-20T16:53:38.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:38.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:38.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:38.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:38.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:38.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:38.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:38.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:38.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:38.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:38.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:38.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:38.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:38.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:38.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6578: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:38.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:38.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:39.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:39.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-20T16:53:39.391 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:53:39.391 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:53:39.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:39.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:39.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:39.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:39.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:39.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:39.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:39.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:39.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:39.498 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:39.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:39.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:39.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:39.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:39.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:39.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:39.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:39.518 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:39.513+0000 7f40ccc40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:39.519 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:39.517+0000 7f40ccc40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:39.520 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:39.517+0000 7f40ccc40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:39.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:39.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:39.663 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:39.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:39.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:39.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:39.948 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:39.945+0000 7f40ccc40a40 -1 Falling back to public interface 2026-02-20T16:53:40.727 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:40.724+0000 7f40ccc40a40 -1 osd.1 161 log_to_monitors true 2026-02-20T16:53:40.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:40.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:40.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:40.823 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:40.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:40.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:40.980 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/3028495264,v1:127.0.0.1:6811/3028495264] [v2:127.0.0.1:6812/3028495264,v1:127.0.0.1:6813/3028495264] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:40.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:40.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:40.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:40.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:40.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:41.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:41.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:41.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:41.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:41.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:41.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:41.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:41.260 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:41.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:41.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:41.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:41.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800072 2026-02-20T16:53:41.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800072 2026-02-20T16:53:41.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800072' 2026-02-20T16:53:41.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:41.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:41.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603842 2026-02-20T16:53:41.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603842 2026-02-20T16:53:41.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800072 1-708669603842' 2026-02-20T16:53:41.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:41.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800072 2026-02-20T16:53:41.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:41.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:41.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800072 2026-02-20T16:53:41.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:41.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800072 2026-02-20T16:53:41.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800072' 2026-02-20T16:53:41.412 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 682899800072 2026-02-20T16:53:41.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:41.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800069 -lt 682899800072 2026-02-20T16:53:41.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:42.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:42.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:42.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800072 -lt 682899800072 2026-02-20T16:53:42.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:42.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603842 2026-02-20T16:53:42.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:42.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:42.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603842 2026-02-20T16:53:42.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:42.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603842 2026-02-20T16:53:42.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2026-02-20T16:53:42.741 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 708669603842 2026-02-20T16:53:42.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:42.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603842 -lt 708669603842 2026-02-20T16:53:42.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:42.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:42.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:43.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:43.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:43.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:43.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:43.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:43.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:43.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:43.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:43.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:43.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-20T16:53:43.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 18 % 2 2026-02-20T16:53:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-20T16:53:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6583: TEST_dual_store_replicated_cluster: local payload=ROBJ18 2026-02-20T16:53:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6584: TEST_dual_store_replicated_cluster: echo ROBJ18 2026-02-20T16:53:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6585: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:43.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:43.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:43.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:43.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:43.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:43.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:43.967 INFO:tasks.workunit.client.0.trial013.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-20T16:53:44.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:44.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:53:44.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:44.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:44.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:44.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:44.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:53:44.081 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:53:44.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:44.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:53:44.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:53:44.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:44.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:44.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:44.098 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:44.092+0000 7fbb542fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:44.099 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:44.096+0000 7fbb542fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:44.100 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:44.096+0000 7fbb542fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:44.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:44.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:44.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:44.544 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:44.540+0000 7fbb542fda40 -1 Falling back to public interface 2026-02-20T16:53:45.076 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:45.072+0000 7fbb542fda40 -1 osd.0 167 log_to_monitors true 2026-02-20T16:53:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:45.404 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:45.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [159,167) [v2:127.0.0.1:6800/2236075478,v1:127.0.0.1:6801/2236075478] [v2:127.0.0.1:6802/2236075478,v1:127.0.0.1:6803/2236075478] exists,up 04816cab-d675-45b4-8f5b-ca3ee92c34e1 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:45.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:45.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:45.564 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:45.564 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:45.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:45.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:45.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:45.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:45.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:45.845 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:45.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:45.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:45.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407618 2026-02-20T16:53:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407618 2026-02-20T16:53:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618' 2026-02-20T16:53:45.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:45.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:45.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603848 2026-02-20T16:53:45.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603848 2026-02-20T16:53:45.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618 1-708669603848' 2026-02-20T16:53:45.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:45.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407618 2026-02-20T16:53:45.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:45.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:45.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407618 2026-02-20T16:53:45.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:45.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407618 2026-02-20T16:53:45.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2026-02-20T16:53:45.995 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 734439407618 2026-02-20T16:53:45.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:46.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 734439407618 2026-02-20T16:53:46.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:47.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:47.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:47.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 734439407618 2026-02-20T16:53:47.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:48.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:48.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:48.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407620 -lt 734439407618 2026-02-20T16:53:48.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:48.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603848 2026-02-20T16:53:48.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:48.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:48.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603848 2026-02-20T16:53:48.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:48.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603848 2026-02-20T16:53:48.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603848' 2026-02-20T16:53:48.485 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 708669603848 2026-02-20T16:53:48.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:48.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603849 -lt 708669603848 2026-02-20T16:53:48.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:48.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:48.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:48.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:48.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:48.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:49.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:49.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:49.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:49.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6586: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:49.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:49.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:49.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:49.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:49.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:49.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-20T16:53:49.691 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:53:49.691 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:49.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:49.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:49.784 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:49.784 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:49.784 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:49.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:49.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:49.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:49.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:49.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:49.809 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:49.804+0000 7f82a2da5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:49.811 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:49.808+0000 7f82a2da5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:49.812 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:49.808+0000 7f82a2da5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:49.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:49.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:49.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:49.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:49.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:50.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:50.208 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:50.204+0000 7f82a2da5a40 -1 Falling back to public interface 2026-02-20T16:53:50.911 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:50.908+0000 7f82a2da5a40 -1 osd.1 172 log_to_monitors true 2026-02-20T16:53:51.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:51.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:51.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:51.110 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:51.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:51.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [165,172) [v2:127.0.0.1:6810/1715559006,v1:127.0.0.1:6811/1715559006] [v2:127.0.0.1:6812/1715559006,v1:127.0.0.1:6813/1715559006] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:51.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:51.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:51.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:51.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:51.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:51.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:51.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:51.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:51.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:51.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:51.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:51.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:51.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:51.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:51.551 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:51.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:51.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:51.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:51.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407625 2026-02-20T16:53:51.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407625 2026-02-20T16:53:51.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407625' 2026-02-20T16:53:51.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:51.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:51.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=755914244098 2026-02-20T16:53:51.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 755914244098 2026-02-20T16:53:51.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407625 1-755914244098' 2026-02-20T16:53:51.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:51.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407625 2026-02-20T16:53:51.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:51.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:51.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407625 2026-02-20T16:53:51.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:51.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407625 2026-02-20T16:53:51.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407625' 2026-02-20T16:53:51.706 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 734439407625 2026-02-20T16:53:51.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:51.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407624 -lt 734439407625 2026-02-20T16:53:51.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:52.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:52.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:53.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407624 -lt 734439407625 2026-02-20T16:53:53.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:54.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:53:54.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:54.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407627 -lt 734439407625 2026-02-20T16:53:54.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:54.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-755914244098 2026-02-20T16:53:54.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:54.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:54.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-755914244098 2026-02-20T16:53:54.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:54.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=755914244098 2026-02-20T16:53:54.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 755914244098' 2026-02-20T16:53:54.199 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 755914244098 2026-02-20T16:53:54.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:54.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 755914244100 -lt 755914244098 2026-02-20T16:53:54.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:54.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:54.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:54.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:54.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:54.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:54.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:54.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:54.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:54.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:54.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:54.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:54.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:54.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:54.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6588: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:53:54.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:53:54.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:53:54.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:53:54.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:53:54.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:53:54.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:53:55.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:55.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-20T16:53:55.395 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-20T16:53:55.395 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-20T16:53:55.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:53:55.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:55.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:53:55.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:53:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:53:55.523 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:55.520+0000 7fe714be0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:55.525 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:55.520+0000 7fe714be0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:55.526 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:55.520+0000 7fe714be0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:53:55.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:55.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:53:55.663 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:53:55.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:55.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:55.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:53:55.988 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:55.984+0000 7fe714be0a40 -1 Falling back to public interface 2026-02-20T16:53:56.618 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:53:56.616+0000 7fe714be0a40 -1 osd.1 177 log_to_monitors true 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:53:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:53:56.973 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [176,177) [v2:127.0.0.1:6810/2930652614,v1:127.0.0.1:6811/2930652614] [v2:127.0.0.1:6812/2930652614,v1:127.0.0.1:6813/2930652614] exists,up 2efb251e-85ef-4aa0-8760-d56d550aa12b 2026-02-20T16:53:56.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:53:56.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:53:56.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:53:56.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:53:56.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:53:56.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:53:56.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:53:56.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:53:56.975 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:53:56.975 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:53:56.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:53:56.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:53:56.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:53:57.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:53:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:53:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:53:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:53:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:53:57.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:53:57.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:53:57.255 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:53:57.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:53:57.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:57.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:53:57.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407632 2026-02-20T16:53:57.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407632 2026-02-20T16:53:57.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407632' 2026-02-20T16:53:57.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:53:57.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:53:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080578 2026-02-20T16:53:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080578 2026-02-20T16:53:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407632 1-777389080578' 2026-02-20T16:53:57.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:57.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407632 2026-02-20T16:53:57.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:57.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:53:57.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407632 2026-02-20T16:53:57.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:57.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407632 2026-02-20T16:53:57.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407632' 2026-02-20T16:53:57.408 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 734439407632 2026-02-20T16:53:57.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:57.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407629 -lt 734439407632 2026-02-20T16:53:57.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:53:58.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:53:58.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:53:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407632 -lt 734439407632 2026-02-20T16:53:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:53:58.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080578 2026-02-20T16:53:58.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:53:58.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:53:58.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080578 2026-02-20T16:53:58.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:53:58.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080578 2026-02-20T16:53:58.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2026-02-20T16:53:58.733 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 777389080578 2026-02-20T16:53:58.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:53:58.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080578 -lt 777389080578 2026-02-20T16:53:58.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:53:58.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:58.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:59.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:53:59.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:53:59.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:53:59.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:53:59.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:53:59.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:53:59.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:53:59.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:53:59.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:53:59.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:53:59.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:53:59.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:53:59.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:53:59.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:53:59.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:53:59.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-20T16:53:59.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-20T16:53:59.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 19 % 2 2026-02-20T16:53:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-20T16:53:59.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-20T16:53:59.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-20T16:53:59.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-20T16:53:59.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-20T16:53:59.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-20T16:53:59.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:53:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-20T16:53:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6599: TEST_dual_store_replicated_cluster: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-20T16:53:59.677 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: {} 2026-02-20T16:53:59.677 INFO:tasks.workunit.client.0.trial013.stderr:osd.0: osd_max_object_size = '' 2026-02-20T16:53:59.686 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: {} 2026-02-20T16:53:59.686 INFO:tasks.workunit.client.0.trial013.stderr:osd.1: osd_max_object_size = '' 2026-02-20T16:53:59.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6601: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:53:59.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-20T16:53:59.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:53:59.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:53:59.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:53:59.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:53:59.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:53:59.858 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:53:59.858 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:53:59.859 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:53:59.859 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:53:59.859 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:59.860 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:59.860 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:53:59.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:53:59.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:53:59.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:53:59.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:53:59.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:53:59.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:53:59.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:53:59.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:53:59.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:53:59.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:53:59.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:53:59.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:53:59.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-20T16:53:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6602: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-20T16:53:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:53:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-20T16:53:59.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:53:59.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-20T16:53:59.987 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-20T16:53:59.987 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:54:00.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:54:00.145 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:54:00.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:54:00.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:54:00.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:54:00.148 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:54:00.148 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:54:00.148 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:54:00.149 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:00.149 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:00.149 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:00.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:54:00.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:54:00.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:54:00.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:54:00.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:54:00.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:54:00.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:54:00.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:54:00.213 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:00.213 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:00.213 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:00.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:54:00.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:54:00.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6603: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:54:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:54:00.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:54:00.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:54:00.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:54:00.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:54:00.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:54:00.442 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:54:00.443 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-20T16:54:00.443 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:54:00.443 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:54:00.444 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:00.444 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:00.444 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:00.444 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:54:00.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:54:00.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:54:00.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:54:00.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:54:00.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-20T16:54:00.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T16:54:00.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T16:54:00.505 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:00.505 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:00.506 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:00.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.1.asok 2026-02-20T16:54:00.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:54:00.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-20T16:54:00.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6606: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-20T16:54:00.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:54:00.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:00.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:00.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:00.564 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:00.564 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:00.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:00.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:00.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:00.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:00.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:00.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:00.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:00.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:00.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:01.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:01.037 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:01.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:01.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:01.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:01.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407636 2026-02-20T16:54:01.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407636 2026-02-20T16:54:01.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407636' 2026-02-20T16:54:01.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:01.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:01.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080583 2026-02-20T16:54:01.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080583 2026-02-20T16:54:01.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407636 1-777389080583' 2026-02-20T16:54:01.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:01.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407636 2026-02-20T16:54:01.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:01.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:01.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407636 2026-02-20T16:54:01.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:01.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407636 2026-02-20T16:54:01.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407636' 2026-02-20T16:54:01.187 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407636 2026-02-20T16:54:01.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:01.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407634 -lt 734439407636 2026-02-20T16:54:01.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:02.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:02.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:02.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407637 -lt 734439407636 2026-02-20T16:54:02.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:02.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080583 2026-02-20T16:54:02.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:02.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:02.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080583 2026-02-20T16:54:02.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:02.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080583 2026-02-20T16:54:02.510 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080583 2026-02-20T16:54:02.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080583' 2026-02-20T16:54:02.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:02.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080583 -lt 777389080583 2026-02-20T16:54:02.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:02.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:02.682 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:02.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:02.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:02.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:02.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:02.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:02.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:02.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:02.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:02.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:02.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:54:03.063 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:03.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:03.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:03.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:03.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:03.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:03.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:03.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:03.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:04.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:04.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:04.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:04.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:04.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:04.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:04.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:04.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:04.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:05.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:05.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:05.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:05.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:05.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:05.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:05.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:05.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:03.673546+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6607: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6608: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:05.562 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:05.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:05.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:05.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:05.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:05.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:06.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:06.022 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:06.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:06.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:06.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:06.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407642 2026-02-20T16:54:06.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407642 2026-02-20T16:54:06.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407642' 2026-02-20T16:54:06.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:06.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:06.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080589 2026-02-20T16:54:06.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080589 2026-02-20T16:54:06.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407642 1-777389080589' 2026-02-20T16:54:06.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:06.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407642 2026-02-20T16:54:06.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:06.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:06.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407642 2026-02-20T16:54:06.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:06.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407642 2026-02-20T16:54:06.183 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407642 2026-02-20T16:54:06.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407642' 2026-02-20T16:54:06.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:06.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407641 -lt 734439407642 2026-02-20T16:54:06.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:07.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:07.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:07.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407641 -lt 734439407642 2026-02-20T16:54:07.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:08.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:54:08.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:08.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407644 -lt 734439407642 2026-02-20T16:54:08.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:08.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080589 2026-02-20T16:54:08.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:08.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:08.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080589 2026-02-20T16:54:08.669 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:08.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080589 2026-02-20T16:54:08.671 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080589 2026-02-20T16:54:08.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080589' 2026-02-20T16:54:08.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:08.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080590 -lt 777389080589 2026-02-20T16:54:08.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:08.828 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:08.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:08.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:08.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:08.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:09.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:09.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:54:09.209 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:09.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:09.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:09.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:09.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:03.673546+0000 '>' 2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:10.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:10.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:10.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:10.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:10.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:10.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:10.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:10.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:03.673546+0000 '>' 2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:10.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:11.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:11.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:11.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:11.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:11.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:11.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:11.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:09.716602+0000 '>' 2026-02-20T16:54:03.673546+0000 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6609: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:11.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:11.691 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:11.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6611: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6614: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:11.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:11.718 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:11.718 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:11.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:11.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:11.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:12.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:12.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:12.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:12.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:12.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:12.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:12.183 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:12.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:12.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407650 2026-02-20T16:54:12.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407650 2026-02-20T16:54:12.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407650' 2026-02-20T16:54:12.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:12.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:12.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080596 2026-02-20T16:54:12.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080596 2026-02-20T16:54:12.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407650 1-777389080596' 2026-02-20T16:54:12.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:12.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407650 2026-02-20T16:54:12.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:12.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:12.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407650 2026-02-20T16:54:12.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:12.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407650 2026-02-20T16:54:12.334 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407650 2026-02-20T16:54:12.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407650' 2026-02-20T16:54:12.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:12.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407648 -lt 734439407650 2026-02-20T16:54:12.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:13.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:13.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:13.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407651 -lt 734439407650 2026-02-20T16:54:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:13.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080596 2026-02-20T16:54:13.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:13.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:13.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080596 2026-02-20T16:54:13.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:13.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080596 2026-02-20T16:54:13.662 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080596 2026-02-20T16:54:13.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080596' 2026-02-20T16:54:13.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:13.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080597 -lt 777389080596 2026-02-20T16:54:13.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:13.830 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:13.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:13.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:54:13.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:13.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:13.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:13.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:13.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:13.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:13.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:13.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:14.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:14.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:54:14.213 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-20T16:54:14.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:52:07.856113+0000 last_deep_scrub_stamp 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:14.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:14.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:14.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:14.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:14.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:15.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:15.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:15.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:15.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:15.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:15.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:15.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:15.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:15.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:16.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:16.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:16.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:16.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:16.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:16.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:16.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:16.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:52:07.856113+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:16.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:17.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:17.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:17.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:17.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:17.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:17.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:17.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:17.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:52:07.856113+0000 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6615: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:17.868 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:17.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:17.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6617: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:17.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6620: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-20T16:54:17.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:54:17.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:17.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:17.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:17.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:17.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:17.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:17.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:17.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:18.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:18.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:18.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:18.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:18.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:18.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:18.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:18.376 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:18.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:18.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:18.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:18.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407657 2026-02-20T16:54:18.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407657 2026-02-20T16:54:18.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407657' 2026-02-20T16:54:18.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:18.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:18.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080603 2026-02-20T16:54:18.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080603 2026-02-20T16:54:18.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407657 1-777389080603' 2026-02-20T16:54:18.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:18.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407657 2026-02-20T16:54:18.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:18.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:18.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407657 2026-02-20T16:54:18.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:18.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407657 2026-02-20T16:54:18.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407657' 2026-02-20T16:54:18.527 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407657 2026-02-20T16:54:18.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:18.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407655 -lt 734439407657 2026-02-20T16:54:18.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:19.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:19.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:19.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407658 -lt 734439407657 2026-02-20T16:54:19.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:19.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080603 2026-02-20T16:54:19.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:19.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:19.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080603 2026-02-20T16:54:19.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:19.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080603 2026-02-20T16:54:19.851 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080603 2026-02-20T16:54:19.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080603' 2026-02-20T16:54:19.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:20.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080604 -lt 777389080603 2026-02-20T16:54:20.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:20.015 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:20.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:20.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:20.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:20.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:20.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:20.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:20.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:54:20.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:20.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:20.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:20.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:20.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:54:20.392 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:54:20.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:20.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:20.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:20.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:20.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:20.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:20.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:21.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:21.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:21.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:21.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:21.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:21.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:21.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:21.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:21.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:22.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:22.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:22.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:22.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:22.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:22.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:22.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:22.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:22.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:23.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:23.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:23.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:23.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:23.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:23.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:23.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:24.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:20.796276+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:24.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:24.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6621: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:24.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:24.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:24.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:24.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6623: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:24.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6626: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dp_results.json td/osd-scrub-repair/sh2_results.json 2026-02-20T16:54:24.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6630: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-20T16:54:24.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-20T16:54:24.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-20T16:54:24.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-20T16:54:24.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-20T16:54:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-20T16:54:24.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-20T16:54:24.241 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-20T16:54:24.241 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:54:24.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.39930/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-20T16:54:24.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-20T16:54:24.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-20T16:54:24.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-20T16:54:24.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-20T16:54:24.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T16:54:24.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T16:54:24.305 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:24.305 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:24.305 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:24.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-osd.0.asok 2026-02-20T16:54:24.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-20T16:54:24.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.39930/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-20T16:54:24.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6632: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-20T16:54:24.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-20T16:54:24.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:24.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:24.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:24.365 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:24.365 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:24.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:24.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:24.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:24.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:24.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:24.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:24.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:24.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:24.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:24.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:24.840 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:24.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:24.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:24.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407664 2026-02-20T16:54:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407664 2026-02-20T16:54:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407664' 2026-02-20T16:54:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:24.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080611 2026-02-20T16:54:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080611 2026-02-20T16:54:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407664 1-777389080611' 2026-02-20T16:54:24.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:24.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407664 2026-02-20T16:54:24.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:24.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:24.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407664 2026-02-20T16:54:24.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:24.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407664 2026-02-20T16:54:24.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407664' 2026-02-20T16:54:24.991 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407664 2026-02-20T16:54:24.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:25.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407662 -lt 734439407664 2026-02-20T16:54:25.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:26.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:26.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:26.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407665 -lt 734439407664 2026-02-20T16:54:26.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:26.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080611 2026-02-20T16:54:26.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:26.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:26.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080611 2026-02-20T16:54:26.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:26.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080611 2026-02-20T16:54:26.313 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080611 2026-02-20T16:54:26.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080611' 2026-02-20T16:54:26.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:26.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080611 -lt 777389080611 2026-02-20T16:54:26.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:26.483 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:26.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:26.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:54:26.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:26.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:26.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:26.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:26.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:26.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:26.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:26.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:26.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:26.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-20T16:54:26.867 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-20T16:54:26.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-20T16:54:14.875143+0000 last_deep_scrub_stamp 2026-02-20T16:54:26.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:26.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:26.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:26.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:26.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:27.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:27.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:28.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:28.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:28.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:28.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:28.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:28.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:28.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:28.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:28.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:29.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:29.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:29.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:29.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:29.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:29.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:29.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:29.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:14.875143+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:29.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:30.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:30.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:30.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-20T16:54:30.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:30.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T16:54:30.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-20T16:54:30.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:30.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:27.798779+0000 '>' 2026-02-20T16:54:14.875143+0000 2026-02-20T16:54:30.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:30.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6634: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:30.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6635: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6637: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:30.517 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:30.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6642: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:30.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-20T16:54:30.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:30.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6646: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6649: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-20T16:54:30.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T16:54:30.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:30.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:30.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:30.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:30.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:30.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:31.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:31.054 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:54:31.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:31.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:31.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:31.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407671 2026-02-20T16:54:31.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407671 2026-02-20T16:54:31.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407671' 2026-02-20T16:54:31.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:31.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:31.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080618 2026-02-20T16:54:31.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080618 2026-02-20T16:54:31.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407671 1-777389080618' 2026-02-20T16:54:31.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:31.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407671 2026-02-20T16:54:31.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:31.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:31.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407671 2026-02-20T16:54:31.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:31.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407671 2026-02-20T16:54:31.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407671' 2026-02-20T16:54:31.203 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 734439407671 2026-02-20T16:54:31.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:31.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407669 -lt 734439407671 2026-02-20T16:54:31.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:32.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:32.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:32.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407672 -lt 734439407671 2026-02-20T16:54:32.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080618 2026-02-20T16:54:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:32.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:32.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080618 2026-02-20T16:54:32.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:32.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080618 2026-02-20T16:54:32.527 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 777389080618 2026-02-20T16:54:32.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080618' 2026-02-20T16:54:32.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080618 -lt 777389080618 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 3.0 loop 0 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-20T16:54:32.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T16:54:32.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-20T16:54:32.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T16:54:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-20T16:54:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T16:54:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:32.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:32.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:32.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-20T16:54:33.067 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:54:33.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:33.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:27.798779+0000 '>' 2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:33.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:34.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:34.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:34.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:34.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:34.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:34.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:34.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:34.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:27.798779+0000 '>' 2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:34.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:54:35.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:54:35.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:54:35.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-20T16:54:35.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-20T16:54:35.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:54:35.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:54:35.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-20T16:54:35.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:33.833337+0000 '>' 2026-02-20T16:54:27.798779+0000 2026-02-20T16:54:35.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:54:35.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6651: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:35.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6652: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6654: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:35.561 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:35.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: . as $in 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: | if type == "object" then 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: reduce keys[] as $key 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: else f 2026-02-20T16:54:35.592 INFO:tasks.workunit.client.0.trial013.stderr: end; 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-20T16:54:35.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-20T16:54:35.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6665: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_w13_results.json td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-20T16:54:35.630 INFO:tasks.workunit.client.0.trial013.stdout:--- td/osd-scrub-repair/dpPart2_w13_results.json 2026-02-20 16:54:30.543657818 +0000 2026-02-20T16:54:35.630 INFO:tasks.workunit.client.0.trial013.stdout:+++ td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-20 16:54:35.587527311 +0000 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:@@ -233,6 +233,38 @@ 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head" 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "selected_object_info": { 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "alloc_hint_flags": 0, 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "data_digest": "0x2ddbf8f5", 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "expected_object_size": 0, 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "expected_write_size": 0, 2026-02-20T16:54:35.631 INFO:tasks.workunit.client.0.trial013.stdout:+ "flags": [ 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "dirty", 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "omap", 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "data_digest", 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "omap_digest" 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ ], 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "lost": 0, 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "manifest": { 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "type": 0 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ }, 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "oid": { 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "hash": 2682806379, 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "key": "", 2026-02-20T16:54:35.632 INFO:tasks.workunit.client.0.trial013.stdout:+ "max": 0, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "namespace": "", 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "oid": "ROBJ13", 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "pool": 3, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "snapid": -2 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ }, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "omap_digest": "0x6441854d", 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "shard_versions": [], 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "size": 7, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "truncate_seq": 0, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "truncate_size": 0, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "user_version": 39, 2026-02-20T16:54:35.633 INFO:tasks.workunit.client.0.trial013.stdout:+ "watchers": {} 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:+ }, 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:@@ -242,17 +274,14 @@ 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:- "errors": [ 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:- "read_error" 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:- ], 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout:+ "errors": [], 2026-02-20T16:54:35.634 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: "size": 7 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout:- "stat_error", 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout:- "read_error" 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout:+ "stat_error" 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:54:35.635 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T16:54:35.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6666: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_wo13_results.json td/osd-scrub-repair/shPart2_wo13_results.json 2026-02-20T16:54:35.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6668: TEST_dual_store_replicated_cluster: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-20T16:54:35.838 INFO:tasks.workunit.client.0.trial013.stderr:pool 'csr_pool' removed 2026-02-20T16:54:35.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6669: TEST_dual_store_replicated_cluster: return 0 2026-02-20T16:54:35.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:54:35.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:54:35.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:54:35.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:54:35.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:54:35.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:54:35.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:54:35.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:54:35.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:54:35.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:54:35.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:54:35.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:54:35.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:54:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:54:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:54:35.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:54:35.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:54:35.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:54:35.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:54:35.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:54:35.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:54:35.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:54:35.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:54:36.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:54:36.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:54:36.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:54:36.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:54:36.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:54:36.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:54:36.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:54:36.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:54:36.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:54:36.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:54:36.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:54:36.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:54:36.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:54:36.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:54:36.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:54:36.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:54:36.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:54:36.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:54:36.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:54:36.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:54:36.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:54:36.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:54:36.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:54:36.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:54:36.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:54:36.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1037: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=false 2026-02-20T16:54:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:54:36.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:54:36.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:54:36.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:36.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:36.046 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:36.047 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.047 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:36.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:54:36.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:54:36.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:54:36.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:54:36.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:54:36.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:54:36.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.079 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:54:36.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:54:36.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:54:36.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:54:36.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:54:36.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:54:36.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:54:36.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:54:36.135 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:54:36.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:54:36.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T16:54:36.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:54:36.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:54:36.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:54:36.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:54:36.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:54:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:54:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:36.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:36.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:36.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.284 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:36.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:54:36.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:54:36.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:54:36.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:36.310 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:54:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:54:36.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:54:36.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:54:36.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:54:36.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:54:36.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=10a1af73-918b-4794-859f-984453c516b5 2026-02-20T16:54:36.313 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 10a1af73-918b-4794-859f-984453c516b5 2026-02-20T16:54:36.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 10a1af73-918b-4794-859f-984453c516b5' 2026-02-20T16:54:36.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:54:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMkZhpXt1QExAAf6B2IX76AN2cYlXfbHghTw== 2026-02-20T16:54:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMkZhpXt1QExAAf6B2IX76AN2cYlXfbHghTw=="}' 2026-02-20T16:54:36.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 10a1af73-918b-4794-859f-984453c516b5 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:54:36.414 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:54:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:54:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMkZhpXt1QExAAf6B2IX76AN2cYlXfbHghTw== --osd-uuid 10a1af73-918b-4794-859f-984453c516b5 2026-02-20T16:54:36.440 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:36.439+0000 7f23facb2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:36.442 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:36.439+0000 7f23facb2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:36.443 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:36.443+0000 7f23facb2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:36.443 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:36.443+0000 7f23facb2a40 -1 bdev(0x555d46f16c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:54:36.443 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:36.443+0000 7f23facb2a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:54:37.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:54:37.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:54:37.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:54:37.212 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:54:37.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:54:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:54:37.314 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:54:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:54:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:54:37.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:54:37.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:54:37.340 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:37.339+0000 7fc4741b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:37.343 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:37.343+0000 7fc4741b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:37.344 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:37.343+0000 7fc4741b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:37.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:54:37.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:54:37.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:54:37.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:54:37.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:54:37.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:37.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:54:37.409 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:54:37.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:37.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:54:37.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:37.744 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:37.743+0000 7fc4741b0a40 -1 Falling back to public interface 2026-02-20T16:54:38.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:38.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:38.499 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:54:38.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:54:38.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:38.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:54:38.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:38.756 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:38.755+0000 7fc4741b0a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:54:39.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:39.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:39.609 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:54:39.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:54:39.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:39.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:54:39.765 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2115044621,v1:127.0.0.1:6801/2115044621] [v2:127.0.0.1:6802/2115044621,v1:127.0.0.1:6803/2115044621] exists,up 10a1af73-918b-4794-859f-984453c516b5 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:54:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:39.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:54:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:54:39.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:54:39.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18a8c795-151b-4a58-adb9-ddcda68c5003 2026-02-20T16:54:39.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 18a8c795-151b-4a58-adb9-ddcda68c5003' 2026-02-20T16:54:39.770 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 18a8c795-151b-4a58-adb9-ddcda68c5003 2026-02-20T16:54:39.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:54:39.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPkZhpvc3FLhAACLvxI5sFOF55bkL7SPXcRQ== 2026-02-20T16:54:39.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPkZhpvc3FLhAACLvxI5sFOF55bkL7SPXcRQ=="}' 2026-02-20T16:54:39.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18a8c795-151b-4a58-adb9-ddcda68c5003 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:54:39.946 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:54:39.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:54:39.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPkZhpvc3FLhAACLvxI5sFOF55bkL7SPXcRQ== --osd-uuid 18a8c795-151b-4a58-adb9-ddcda68c5003 2026-02-20T16:54:39.972 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:39.971+0000 7f8be285da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:39.973 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:39.971+0000 7f8be285da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:39.974 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:39.975+0000 7f8be285da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:39.975 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:39.975+0000 7f8be285da40 -1 bdev(0x5602f8412c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:54:39.975 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:39.975+0000 7f8be285da40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:54:40.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:54:40.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:54:40.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:54:40.659 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:54:40.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:54:40.856 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:54:40.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:54:40.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:54:40.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:54:40.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:54:40.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:54:40.879 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:40.879+0000 7f8f1a479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:40.881 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:40.879+0000 7f8f1a479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:40.882 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:40.879+0000 7f8f1a479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:41.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:54:41.024 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:54:41.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:41.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:54:41.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:41.264 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:41.263+0000 7f8f1a479a40 -1 Falling back to public interface 2026-02-20T16:54:41.852 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:41.851+0000 7f8f1a479a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:54:42.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:42.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:42.184 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:54:42.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:54:42.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:42.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:54:42.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:43.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:43.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:43.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:54:43.344 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:54:43.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:43.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:54:43.502 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3581984166,v1:127.0.0.1:6811/3581984166] [v2:127.0.0.1:6812/3581984166,v1:127.0.0.1:6813/3581984166] exists,up 18a8c795-151b-4a58-adb9-ddcda68c5003 2026-02-20T16:54:43.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:54:43.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:43.507 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:54:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:54:43.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:54:43.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:54:43.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:54:43.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:54:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4bd008e7-de90-4556-8a18-8bde9546da40 2026-02-20T16:54:43.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4bd008e7-de90-4556-8a18-8bde9546da40' 2026-02-20T16:54:43.510 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 4bd008e7-de90-4556-8a18-8bde9546da40 2026-02-20T16:54:43.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:54:43.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTkZhpYgAOHxAAI4oX5o5dm44Tf55VBdBmFg== 2026-02-20T16:54:43.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTkZhpYgAOHxAAI4oX5o5dm44Tf55VBdBmFg=="}' 2026-02-20T16:54:43.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4bd008e7-de90-4556-8a18-8bde9546da40 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:54:43.675 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:54:43.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:54:43.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTkZhpYgAOHxAAI4oX5o5dm44Tf55VBdBmFg== --osd-uuid 4bd008e7-de90-4556-8a18-8bde9546da40 2026-02-20T16:54:43.701 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:43.699+0000 7fa678a35a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:43.702 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:43.703+0000 7fa678a35a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:43.703 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:43.703+0000 7fa678a35a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:43.704 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:43.703+0000 7fa678a35a40 -1 bdev(0x559d35c7cc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:54:43.704 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:43.703+0000 7fa678a35a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:54:44.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:54:44.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:54:44.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:54:44.351 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:54:44.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:54:44.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:54:44.546 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:54:44.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:54:44.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:54:44.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:54:44.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:54:44.561 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:44.559+0000 7f3013e42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:44.562 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:44.559+0000 7f3013e42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:44.563 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:44.563+0000 7f3013e42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:44.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:54:44.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:54:44.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:54:44.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:54:44.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:54:44.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:44.708 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:54:44.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:54:44.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:44.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:54:44.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:44.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:44.975+0000 7f3013e42a40 -1 Falling back to public interface 2026-02-20T16:54:45.579 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:45.575+0000 7f3013e42a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:54:45.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:45.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:45.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:54:45.865 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:54:45.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:45.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:54:46.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:47.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:54:47.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:47.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:54:47.029 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:54:47.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:47.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:54:47.186 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1996928210,v1:127.0.0.1:6819/1996928210] [v2:127.0.0.1:6820/1996928210,v1:127.0.0.1:6821/1996928210] exists,up 4bd008e7-de90-4556-8a18-8bde9546da40 2026-02-20T16:54:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:54:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:54:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:54:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-20T16:54:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:54:47.334 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:54:47.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:54:47.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:54:47.532 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:54:47.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:54:48.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:54:48.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-20T16:54:48.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:54:48.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:54:48.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:54:48.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:54:48.825 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:48.825 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:48.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:48.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:48.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:48.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:54:48.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:54:48.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:54:48.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:54:48.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:48.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:49.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:49.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:54:49.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:54:49.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:54:49.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:49.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:49.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:54:49.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:54:49.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-20T16:54:49.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:49.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:54:49.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509442 2026-02-20T16:54:49.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509442 2026-02-20T16:54:49.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-64424509442' 2026-02-20T16:54:49.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:49.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:54:49.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:49.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:49.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:54:49.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:49.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:54:49.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:54:49.335 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:54:49.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:54:49.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:50.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:50.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:50.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:54:50.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:50.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:54:50.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:50.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:50.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:54:50.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:50.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:54:50.670 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:54:50.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:54:50.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:50.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T16:54:50.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:50.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509442 2026-02-20T16:54:50.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:50.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:54:50.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509442 2026-02-20T16:54:50.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:50.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509442 2026-02-20T16:54:50.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2026-02-20T16:54:50.842 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509442 2026-02-20T16:54:50.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:54:51.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509442 2026-02-20T16:54:51.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:54:51.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:54:51.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:54:51.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:54:51.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:54:51.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:54:51.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:54:51.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:54:51.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:54:51.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:54:51.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:54:51.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:54:51.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:54:51.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:54:51.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:54:51.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:54:51.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:54:51.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-20T16:54:51.765 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:54:51.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:54:51.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:54:52.072 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:54:52.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:54:53.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:54:53.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:54:53.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:54:53.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:54:53.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:54:53.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:54:53.089 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:54:53.090 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:54:53.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:54:53.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:54:53.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:54:53.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:54:53.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:54:53.366 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:54:53.366 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:54:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:54:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:53.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:54:53.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:54:53.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:54:53.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:54:53.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:53.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:54:53.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-20T16:54:53.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-20T16:54:53.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-20T16:54:53.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:54:53.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:54:53.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-20T16:54:53.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-20T16:54:53.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444' 2026-02-20T16:54:53.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:53.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:54:53.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:53.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:54:53.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:54:53.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:53.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:54:53.590 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:54:53.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:54:53.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:53.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:54:53.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:54:54.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:54:54.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:54:54.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:54:54.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:54.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-20T16:54:54.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:54.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:54:54.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-20T16:54:54.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:54.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-20T16:54:54.926 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672965 2026-02-20T16:54:54.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-20T16:54:54.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:54:55.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-20T16:54:55.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:54:55.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-20T16:54:55.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:54:55.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:54:55.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-20T16:54:55.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:54:55.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-20T16:54:55.093 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509444 2026-02-20T16:54:55.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-20T16:54:55.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:54:55.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-20T16:54:55.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:54:55.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:54:55.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:54:55.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:54:55.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:54:55.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:54:55.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:54:55.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:54:55.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:54:55.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:54:55.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:54:55.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:54:55.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:54:55.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:54:55.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:54:55.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:54:55.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:54:55.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:54:56.001 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:54:56.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:54:56.206 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:54:56.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:54:56.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:54:56.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:54:56.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-20T16:54:56.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:54:56.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-20T16:54:56.247 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-20T16:54:56.247 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:54:56.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-20T16:54:56.410 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:54:56.410 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:54:56.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:54:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:54:56.613 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:54:56.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:54:56.818 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:54:56.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:54:56.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:54:56.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:54:56.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-20T16:54:56.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:54:56.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-20T16:54:56.854 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-20T16:54:56.854 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:54:57.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-20T16:54:57.012 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:54:57.012 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:54:57.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-20T16:54:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-20T16:54:57.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-20T16:54:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-20T16:54:57.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:57.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:54:57.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:54:57.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:54:57.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:54:57.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:54:57.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:54:57.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:54:57.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:57.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:54:57.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:54:57.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:54:57.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:54:57.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:54:57.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:54:57.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:54:57.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:54:57.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:54:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:54:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-20T16:54:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-20T16:54:58.127 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-20T16:54:58.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-20T16:54:58.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-20T16:54:58.451 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-20T16:54:58.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-20T16:54:58.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-20T16:54:58.765 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-20T16:54:58.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-20T16:54:58.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-20T16:54:59.067 INFO:tasks.workunit.client.0.trial013.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-20T16:54:59.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-20T16:54:59.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:54:59.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-20T16:54:59.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:54:59.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:54:59.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:54:59.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:54:59.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:54:59.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:54:59.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:54:59.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:54:59.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:54:59.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:54:59.181 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:59.178+0000 7f3eeef33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:59.183 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:59.182+0000 7f3eeef33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:59.184 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:59.182+0000 7f3eeef33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:54:59.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:54:59.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:54:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:54:59.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:54:59.652 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:54:59.650+0000 7f3eeef33a40 -1 Falling back to public interface 2026-02-20T16:55:00.311 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:00.306+0000 7f3eeef33a40 -1 osd.0 31 log_to_monitors true 2026-02-20T16:55:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:00.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:00.476 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:00.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:00.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6800/904696941,v1:127.0.0.1:6801/904696941] [v2:127.0.0.1:6802/904696941,v1:127.0.0.1:6803/904696941] exists,up 10a1af73-918b-4794-859f-984453c516b5 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:55:00.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:55:00.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:00.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:55:00.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:00.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:00.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:00.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:55:00.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:55:00.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:55:00.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:55:00.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:55:00.651 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:00.650+0000 7fa29a51fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:00.653 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:00.650+0000 7fa29a51fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:00.654 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:00.650+0000 7fa29a51fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:00.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:00.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:01.012 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:01.010+0000 7fa29a51fa40 -1 Falling back to public interface 2026-02-20T16:55:01.742 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:01.738+0000 7fa29a51fa40 -1 osd.1 33 log_to_monitors true 2026-02-20T16:55:01.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:01.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:01.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:01.960 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:01.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:01.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:02.119 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 34 last_clean_interval [10,33) [v2:127.0.0.1:6810/3058818961,v1:127.0.0.1:6811/3058818961] [v2:127.0.0.1:6812/3058818961,v1:127.0.0.1:6813/3058818961] exists,up 18a8c795-151b-4a58-adb9-ddcda68c5003 2026-02-20T16:55:02.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:55:02.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:02.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:02.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:55:02.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:55:02.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:55:02.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:55:02.139 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:02.138+0000 7f060b84da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:02.140 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:02.138+0000 7f060b84da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:02.141 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:02.138+0000 7f060b84da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:02.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:02.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:02.283 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:55:02.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:02.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:02.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:02.556 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:02.554+0000 7f060b84da40 -1 Falling back to public interface 2026-02-20T16:55:03.269 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:03.266+0000 7f060b84da40 -1 osd.2 35 log_to_monitors true 2026-02-20T16:55:03.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:03.440 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:03.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:03.595 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 36 last_clean_interval [15,35) [v2:127.0.0.1:6818/3681457715,v1:127.0.0.1:6819/3681457715] [v2:127.0.0.1:6820/3681457715,v1:127.0.0.1:6821/3681457715] exists,up 4bd008e7-de90-4556-8a18-8bde9546da40 2026-02-20T16:55:03.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:03.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:03.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:03.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-20T16:55:03.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:55:03.794 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' removed 2026-02-20T16:55:03.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:55:03.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:55:03.998 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:55:04.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:55:05.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:55:05.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-20T16:55:05.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:55:05.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:55:05.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:55:05.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:55:05.294 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:55:05.294 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:55:05.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:55:05.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:55:05.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:55:05.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:55:05.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:55:05.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:55:05.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:55:05.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:55:05.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:05.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:55:05.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757251 2026-02-20T16:55:05.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757251 2026-02-20T16:55:05.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251' 2026-02-20T16:55:05.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:05.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:55:05.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659138 2026-02-20T16:55:05.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659138 2026-02-20T16:55:05.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-176093659138' 2026-02-20T16:55:05.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:05.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:55:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-20T16:55:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-20T16:55:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-176093659138 2-193273528322' 2026-02-20T16:55:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:05.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-163208757251 2026-02-20T16:55:05.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:05.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:55:05.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-163208757251 2026-02-20T16:55:05.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:05.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757251 2026-02-20T16:55:05.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 163208757251' 2026-02-20T16:55:05.803 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 163208757251 2026-02-20T16:55:05.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:05.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 163208757251 2026-02-20T16:55:05.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:55:06.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:55:06.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:07.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757251 -lt 163208757251 2026-02-20T16:55:07.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:07.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-176093659138 2026-02-20T16:55:07.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:07.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:55:07.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-176093659138 2026-02-20T16:55:07.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659138 2026-02-20T16:55:07.134 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 176093659138 2026-02-20T16:55:07.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2026-02-20T16:55:07.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:55:07.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659138 -lt 176093659138 2026-02-20T16:55:07.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:07.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-193273528322 2026-02-20T16:55:07.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:07.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:55:07.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-193273528322 2026-02-20T16:55:07.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:07.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-20T16:55:07.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 193273528322' 2026-02-20T16:55:07.304 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 193273528322 2026-02-20T16:55:07.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:55:07.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-20T16:55:07.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:55:07.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:07.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:07.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:55:07.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:55:07.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:55:07.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:55:07.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:55:07.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:55:07.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:55:07.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:55:07.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:55:07.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:55:07.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:07.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:55:08.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-20T16:55:08.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-20T16:55:08.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:08.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:08.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:08.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:08.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:08.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:55:08.484 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:08.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:08.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:08.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:08.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:08.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:08.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:08.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:09.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:09.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:09.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:09.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:09.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:09.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:09.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:09.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:09.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:10.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:10.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:10.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:10.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:10.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:10.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:10.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:10.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:10.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:11.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:11.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:11.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:11.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:12.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:12.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:13.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:13.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:13.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:13.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:13.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:13.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:13.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:13.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:13.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:14.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:14.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:14.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:14.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:14.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:14.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:14.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:14.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:54:52.071087+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:14.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:15.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:15.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:15.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:15.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:15.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:15.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:15.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:15.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:09.386239+0000 '>' 2026-02-20T16:54:52.071087+0000 2026-02-20T16:55:15.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:55:15.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-20T16:55:15.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-20T16:55:15.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-20T16:55:15.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-20T16:55:15.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-20T16:55:15.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:15.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:15.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:15.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:15.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:15.812 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:15.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:15.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:15.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:15.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:15.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:15.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:15.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:15.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:15.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:15.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:15.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:15.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:15.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:15.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:15.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:15.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:15.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:15.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:15.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:15.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:15.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:15.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:15.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:15.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:15.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:15.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:15.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:15.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:15.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:15.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:15.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:15.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:15.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:15.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:15.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:15.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:15.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:15.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:55:15.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:55:15.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:55:15.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:55:15.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:55:15.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:55:15.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1041: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=true 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:55:15.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:55:15.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:55:15.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:15.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:15.880 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:15.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:15.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:55:15.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:55:15.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:55:15.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:55:15.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:55:15.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:55:15.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:55:15.920 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:55:15.920 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:55:15.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:55:15.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:55:15.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:55:15.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:55:15.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:55:15.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:55:15.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:55:15.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:55:15.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:55:15.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:55:16.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:55:16.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:16.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:16.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:16.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:55:16.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:55:16.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-20T16:55:16.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:16.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-20T16:55:16.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:16.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:55:16.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:16.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:16.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:16.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:55:16.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:55:16.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:55:16.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d469ff7-7817-41c1-81c4-c8a253a4a7da 2026-02-20T16:55:16.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5d469ff7-7817-41c1-81c4-c8a253a4a7da' 2026-02-20T16:55:16.158 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 5d469ff7-7817-41c1-81c4-c8a253a4a7da 2026-02-20T16:55:16.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:55:16.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0kZhpj3gLChAA7CaowmLYRl5Wx2nJuWDedw== 2026-02-20T16:55:16.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0kZhpj3gLChAA7CaowmLYRl5Wx2nJuWDedw=="}' 2026-02-20T16:55:16.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d469ff7-7817-41c1-81c4-c8a253a4a7da -i td/osd-scrub-repair/0/new.json 2026-02-20T16:55:16.257 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:55:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0kZhpj3gLChAA7CaowmLYRl5Wx2nJuWDedw== --osd-uuid 5d469ff7-7817-41c1-81c4-c8a253a4a7da 2026-02-20T16:55:16.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:16.278+0000 7f889b282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:16.283 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:16.282+0000 7f889b282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:16.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:16.282+0000 7f889b282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:16.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:16.282+0000 7f889b282a40 -1 bdev(0x55f7c6c98c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:55:16.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:16.282+0000 7f889b282a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:55:17.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:55:17.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:55:17.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:55:17.055 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:55:17.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:55:17.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:55:17.158 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:55:17.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:17.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:55:17.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:55:17.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:55:17.174 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:17.170+0000 7f0ea0c92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:17.175 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:17.174+0000 7f0ea0c92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:17.176 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:17.174+0000 7f0ea0c92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:17.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:55:17.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:17.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:55:17.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:17.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:17.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:17.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:17.254 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:17.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:17.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:17.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:17.596 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:17.594+0000 7f0ea0c92a40 -1 Falling back to public interface 2026-02-20T16:55:18.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:18.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:18.349 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:18.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:18.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:18.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:18.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:18.553 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:18.550+0000 7f0ea0c92a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:55:19.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:19.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:19.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:55:19.440 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:55:19.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:19.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:19.603 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1177948787,v1:127.0.0.1:6801/1177948787] [v2:127.0.0.1:6802/1177948787,v1:127.0.0.1:6803/1177948787] exists,up 5d469ff7-7817-41c1-81c4-c8a253a4a7da 2026-02-20T16:55:19.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:19.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:19.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:19.608 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:55:19.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:55:19.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:55:19.610 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:55:19.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d 2026-02-20T16:55:19.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d' 2026-02-20T16:55:19.611 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d 2026-02-20T16:55:19.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:55:19.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD3kZhpQE0QJRAAWXhQRbqyC9g9+1EALV/7RQ== 2026-02-20T16:55:19.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD3kZhpQE0QJRAAWXhQRbqyC9g9+1EALV/7RQ=="}' 2026-02-20T16:55:19.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d -i td/osd-scrub-repair/1/new.json 2026-02-20T16:55:19.785 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:19.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:55:19.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3kZhpQE0QJRAAWXhQRbqyC9g9+1EALV/7RQ== --osd-uuid c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d 2026-02-20T16:55:19.813 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:19.810+0000 7fbfc23a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:19.815 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:19.814+0000 7fbfc23a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:19.816 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:19.814+0000 7fbfc23a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:19.816 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:19.814+0000 7fbfc23a6a40 -1 bdev(0x55cd33d32c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:55:19.816 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:19.814+0000 7fbfc23a6a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:55:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:55:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:55:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:55:20.472 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:55:20.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:55:20.676 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:55:20.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:55:20.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:20.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:55:20.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:55:20.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:55:20.693 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:20.690+0000 7fb62b9b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:20.694 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:20.690+0000 7fb62b9b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:20.695 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:20.694+0000 7fb62b9b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:20.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:20.832 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:20.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:20.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:20.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:21.120 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:21.118+0000 7fb62b9b9a40 -1 Falling back to public interface 2026-02-20T16:55:21.733 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:21.730+0000 7fb62b9b9a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:55:21.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:21.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:21.993 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:21.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:21.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:21.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:22.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:23.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:23.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:23.152 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:55:23.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:55:23.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:23.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:23.314 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/289389709,v1:127.0.0.1:6811/289389709] [v2:127.0.0.1:6812/289389709,v1:127.0.0.1:6813/289389709] exists,up c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d 2026-02-20T16:55:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:23.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:55:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:23.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:23.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=364e4c6a-8e01-4a17-932f-307b536d97b7 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 364e4c6a-8e01-4a17-932f-307b536d97b7' 2026-02-20T16:55:23.321 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 364e4c6a-8e01-4a17-932f-307b536d97b7 2026-02-20T16:55:23.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:55:23.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7kZhpHtjZExAA8vrqEv2lUjUJAUdbI0ETvg== 2026-02-20T16:55:23.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7kZhpHtjZExAA8vrqEv2lUjUJAUdbI0ETvg=="}' 2026-02-20T16:55:23.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 364e4c6a-8e01-4a17-932f-307b536d97b7 -i td/osd-scrub-repair/2/new.json 2026-02-20T16:55:23.497 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:55:23.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:55:23.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7kZhpHtjZExAA8vrqEv2lUjUJAUdbI0ETvg== --osd-uuid 364e4c6a-8e01-4a17-932f-307b536d97b7 2026-02-20T16:55:23.524 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:23.522+0000 7f02cc356a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:23.525 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:23.522+0000 7f02cc356a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:23.526 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:23.522+0000 7f02cc356a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:23.527 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:23.522+0000 7f02cc356a40 -1 bdev(0x55ebaac7cc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:55:23.527 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:23.526+0000 7f02cc356a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:55:24.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:55:24.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:55:24.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:55:24.119 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:55:24.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:55:24.321 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:55:24.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:55:24.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:24.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:55:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:55:24.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:55:24.337 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:24.334+0000 7fd7b94d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:24.339 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:24.338+0000 7fd7b94d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:24.340 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:24.338+0000 7fd7b94d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:24.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:24.481 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:24.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:24.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:24.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:24.728 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:24.726+0000 7fd7b94d8a40 -1 Falling back to public interface 2026-02-20T16:55:25.340 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:25.338+0000 7fd7b94d8a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:25.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:25.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:26.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:26.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:26.801 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:55:26.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:55:26.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:26.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:26.962 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3906656513,v1:127.0.0.1:6819/3906656513] [v2:127.0.0.1:6820/3906656513,v1:127.0.0.1:6821/3906656513] exists,up 364e4c6a-8e01-4a17-932f-307b536d97b7 2026-02-20T16:55:26.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:26.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:26.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:26.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-20T16:55:26.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:55:27.110 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:55:27.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:55:27.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:55:27.339 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:55:27.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:55:28.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:55:28.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-20T16:55:28.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:55:28.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:55:28.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:55:28.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:55:28.633 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:55:28.633 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:55:28.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:55:28.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:55:28.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:55:28.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:55:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:55:28.919 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:28.919 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:55:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:55:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:28.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:55:28.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:55:28.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:55:28.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:55:28.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:28.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:55:29.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-20T16:55:29.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-20T16:55:29.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-20T16:55:29.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:29.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:55:29.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-20T16:55:29.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-20T16:55:29.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-20T16:55:29.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:29.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:55:29.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:29.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:55:29.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:55:29.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:29.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:55:29.146 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:55:29.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:55:29.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:29.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:55:29.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:55:30.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:55:30.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:30.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:55:30.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:30.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-20T16:55:30.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:30.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:55:30.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-20T16:55:30.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:30.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-20T16:55:30.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-20T16:55:30.484 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705667 2026-02-20T16:55:30.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:55:30.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-20T16:55:30.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:30.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-20T16:55:30.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:30.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:55:30.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-20T16:55:30.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:30.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-20T16:55:30.646 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574850 2026-02-20T16:55:30.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-20T16:55:30.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:55:30.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-20T16:55:30.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:55:30.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:30.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:30.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:55:30.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:55:31.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:55:31.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:55:31.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:31.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:55:31.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T16:55:31.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:55:31.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-20T16:55:31.546 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:55:31.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T16:55:31.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T16:55:31.852 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T16:55:31.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:55:32.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T16:55:32.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T16:55:33.067 INFO:tasks.workunit.client.0.trial013.stderr:set pool 2 allow_ec_overwrites to true 2026-02-20T16:55:33.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:55:33.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:55:33.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:55:33.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:55:33.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:55:33.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:55:33.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:55:33.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:55:33.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:55:33.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:55:33.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:55:33.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:55:33.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:55:33.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:55:33.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:55:33.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:55:33.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:55:33.368 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:33.368 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:55:33.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:55:33.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:33.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:55:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:55:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:55:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:55:33.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:33.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:55:33.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:55:33.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:55:33.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-20T16:55:33.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:33.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:55:33.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-20T16:55:33.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-20T16:55:33.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-55834574852' 2026-02-20T16:55:33.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:33.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:55:33.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:33.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:55:33.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:55:33.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:33.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:55:33.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:55:33.596 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:55:33.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:33.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:55:33.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:55:34.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:55:34.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:34.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-20T16:55:34.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:34.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:55:34.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:34.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:55:34.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:55:34.919 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:34.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:55:34.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:55:34.922 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:55:34.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:55:35.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T16:55:35.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:35.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-20T16:55:35.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:35.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:55:35.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-20T16:55:35.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:35.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-20T16:55:35.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-20T16:55:35.098 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574852 2026-02-20T16:55:35.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:55:35.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-20T16:55:35.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:55:35.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:35.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:35.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:55:35.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:55:35.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:55:35.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:55:35.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:55:35.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:35.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:55:35.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:55:36.025 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:55:36.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:55:36.228 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:55:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:55:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:55:36.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:55:36.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-20T16:55:36.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:55:36.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-20T16:55:36.261 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-20T16:55:36.262 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-20T16:55:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:55:36.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T16:55:36.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-20T16:55:36.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:55:36.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:55:36.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:55:36.625 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:55:36.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:55:36.831 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:55:36.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:55:36.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:55:36.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:55:36.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-20T16:55:36.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T16:55:36.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-20T16:55:36.869 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-20T16:55:36.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T16:55:37.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-20T16:55:37.030 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:37.030 INFO:tasks.workunit.client.0.trial013.stderr:0' 2026-02-20T16:55:37.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-20T16:55:37.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-20T16:55:37.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-20T16:55:37.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-20T16:55:37.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:37.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:55:37.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:37.033 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:37.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:37.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:37.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:37.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:37.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:37.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:55:37.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:37.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:37.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:37.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:37.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:37.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:37.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:37.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T16:55:37.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:37.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:37.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:37.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:37.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:37.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:37.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-20T16:55:37.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-20T16:55:38.144 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-20T16:55:38.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-20T16:55:38.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-20T16:55:38.468 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-20T16:55:38.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-20T16:55:38.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-20T16:55:38.789 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-20T16:55:38.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-20T16:55:38.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-20T16:55:39.075 INFO:tasks.workunit.client.0.trial013.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-20T16:55:39.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-20T16:55:39.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:39.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-20T16:55:39.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:39.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:55:39.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:39.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:39.196 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:39.196 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:39.196 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:39.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:39.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:55:39.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:55:39.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:55:39.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:55:39.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:55:39.221 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:39.217+0000 7f2c25ba3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:39.222 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:39.217+0000 7f2c25ba3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:39.223 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:39.221+0000 7f2c25ba3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:39.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:55:39.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:39.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:55:39.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:39.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:39.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:39.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:39.361 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:55:39.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:39.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:39.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:39.568 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:39.565+0000 7f2c25ba3a40 -1 Falling back to public interface 2026-02-20T16:55:40.372 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:40.369+0000 7f2c25ba3a40 -1 osd.0 31 log_to_monitors true 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:40.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:40.682 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6800/2570434460,v1:127.0.0.1:6801/2570434460] [v2:127.0.0.1:6802/2570434460,v1:127.0.0.1:6803/2570434460] exists,up 5d469ff7-7817-41c1-81c4-c8a253a4a7da 2026-02-20T16:55:40.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:40.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:40.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:40.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:40.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:55:40.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:55:40.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:55:40.699 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:40.697+0000 7f35f65f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:40.700 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:40.697+0000 7f35f65f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:40.701 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:40.697+0000 7f35f65f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:40.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:40.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:41.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:41.100 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:41.097+0000 7f35f65f9a40 -1 Falling back to public interface 2026-02-20T16:55:41.720 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:41.717+0000 7f35f65f9a40 -1 osd.1 33 log_to_monitors true 2026-02-20T16:55:42.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:42.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:42.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:42.007 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:42.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:42.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 34 last_clean_interval [9,33) [v2:127.0.0.1:6810/1736837390,v1:127.0.0.1:6811/1736837390] [v2:127.0.0.1:6812/1736837390,v1:127.0.0.1:6813/1736837390] exists,up c61d8dbf-b3ee-48ed-aeaf-0ef6fe45ec6d 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:55:42.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:42.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:42.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:55:42.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:55:42.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:55:42.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T16:55:42.176 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T16:55:42.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:55:42.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T16:55:42.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T16:55:42.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:55:42.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:55:42.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:55:42.196 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:42.193+0000 7fb76cfe9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:42.198 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:42.193+0000 7fb76cfe9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:42.199 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:42.197+0000 7fb76cfe9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:42.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T16:55:42.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:42.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:42.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:42.596 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:42.593+0000 7fb76cfe9a40 -1 Falling back to public interface 2026-02-20T16:55:43.228 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:43.225+0000 7fb76cfe9a40 -1 osd.2 36 log_to_monitors true 2026-02-20T16:55:43.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:43.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:43.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:43.495 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:43.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:43.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:55:43.650 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 48 up_thru 48 down_at 36 last_clean_interval [13,36) [v2:127.0.0.1:6818/2506032817,v1:127.0.0.1:6819/2506032817] [v2:127.0.0.1:6820/2506032817,v1:127.0.0.1:6821/2506032817] exists,up 364e4c6a-8e01-4a17-932f-307b536d97b7 2026-02-20T16:55:43.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:43.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:43.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:43.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-20T16:55:43.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:55:43.813 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' removed 2026-02-20T16:55:43.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:55:43.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:55:44.016 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:55:44.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:55:45.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:55:45.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-20T16:55:45.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:55:45.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:55:45.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:55:45.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:55:45.311 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:55:45.311 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:55:45.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:55:45.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:55:45.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:55:45.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:55:45.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:55:45.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:55:45.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:55:45.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:55:45.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:55:45.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:55:45.594 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:55:45.594 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:55:45.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:55:45.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:45.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:55:45.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724547 2026-02-20T16:55:45.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724547 2026-02-20T16:55:45.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547' 2026-02-20T16:55:45.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:45.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:55:45.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-20T16:55:45.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-20T16:55:45.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730' 2026-02-20T16:55:45.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:55:45.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:55:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430210 2026-02-20T16:55:45.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430210 2026-02-20T16:55:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730 2-206158430210' 2026-02-20T16:55:45.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:45.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-167503724547 2026-02-20T16:55:45.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:45.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:55:45.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-167503724547 2026-02-20T16:55:45.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724547 2026-02-20T16:55:45.821 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 167503724547 2026-02-20T16:55:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2026-02-20T16:55:45.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:45.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 167503724547 2026-02-20T16:55:45.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:55:46.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:55:46.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:55:47.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724547 -lt 167503724547 2026-02-20T16:55:47.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:47.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-184683593730 2026-02-20T16:55:47.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:47.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:55:47.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-184683593730 2026-02-20T16:55:47.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:47.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-20T16:55:47.155 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 184683593730 2026-02-20T16:55:47.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2026-02-20T16:55:47.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:55:47.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-20T16:55:47.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:55:47.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-206158430210 2026-02-20T16:55:47.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:55:47.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:55:47.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-206158430210 2026-02-20T16:55:47.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:55:47.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=206158430210 2026-02-20T16:55:47.320 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 206158430210 2026-02-20T16:55:47.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 206158430210' 2026-02-20T16:55:47.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:55:47.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 206158430210 -lt 206158430210 2026-02-20T16:55:47.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:55:47.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:47.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:47.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:55:47.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:55:47.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:55:47.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:55:47.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:55:47.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:55:47.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:55:48.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:55:48.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:55:48.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:55:48.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-20T16:55:48.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T16:55:48.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-20T16:55:48.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-20T16:55:48.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:55:48.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-20T16:55:48.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-20T16:55:48.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-20T16:55:48.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-20T16:55:48.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:48.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:48.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:48.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:48.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:48.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-20T16:55:48.489 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-20T16:55:48.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:48.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-20T16:55:48.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:48.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:55:48.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:48.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:48.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:31.852742+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:48.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:49.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:49.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:49.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:49.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:49.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:49.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:49.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:49.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:31.852742+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:49.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:50.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:50.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:50.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:50.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:50.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:50.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:50.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:50.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:31.852742+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:50.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:51.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:51.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:51.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:52.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:31.852742+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:52.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:53.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:53.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:53.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:53.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:53.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:53.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:53.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:53.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:31.852742+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:53.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:55:54.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:55:54.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:55:54.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-20T16:55:54.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-20T16:55:54.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:55:54.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:55:54.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-20T16:55:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:55:49.294925+0000 '>' 2026-02-20T16:55:31.852742+0000 2026-02-20T16:55:54.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:55:54.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-20T16:55:54.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-20T16:55:54.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-20T16:55:54.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-20T16:55:54.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-20T16:55:54.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-20T16:55:54.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-20T16:55:54.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-20T16:55:54.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-20T16:55:54.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:55:54.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:54.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:54.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:54.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:54.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:54.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:54.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:54.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:54.862 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:54.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:54.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:54.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:54.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:54.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:54.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:54.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:54.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:54.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:54.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:54.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:54.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:54.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:54.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:54.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:54.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:54.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:54.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:54.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:54.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:54.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:54.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:54.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:54.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:54.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:54.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:54.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:54.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:54.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:54.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:54.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:54.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:54.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:54.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:54.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:54.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:55:54.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:55:54.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:55:54.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:55:54.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:55:54.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:55:54.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5926: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_periodic_scrub_replicated: local poolname=psr_pool 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5928: TEST_periodic_scrub_replicated: local objname=POBJ 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5929: TEST_periodic_scrub_replicated: return 0 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:54.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:54.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:54.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:54.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:54.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:54.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:54.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:54.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:54.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:54.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:54.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:54.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:54.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:54.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:54.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:54.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:54.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:54.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:54.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:54.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:54.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:54.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:55:54.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:55:54.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:55:54.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:55:54.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:55:54.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:55:54.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:55:54.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:55:54.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:55:54.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:55:54.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:55:54.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:55:54.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:55:54.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:55:54.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:55:54.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:55:54.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:55:54.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:55:54.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:55:54.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:55:54.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:55:54.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:55:54.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:55:54.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:55:54.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:55:54.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:55:54.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: local poolname=testpool 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: local OSDS=2 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: local OBJS=30 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: local REPAIRS=20 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: TEST_repair_stats: run_mon td/osd-scrub-repair a 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:55:54.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:55:54.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:55:54.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:55:54.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:54.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:54.964 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:54.964 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:54.965 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:54.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:54.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:55:55.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:55:55.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:55:55.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:55:55.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:55:55.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:55:55.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:55:55.005 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:55:55.005 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:55:55.005 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:55:55.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:55:55.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:55:55.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:55:55.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:55:55.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:55:55.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:55:55.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:55:55.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:55:55.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:55:55.062 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:55:55.062 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:55.063 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:55.063 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:55:55.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:55:55.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:55:55.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:722: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2026-02-20T16:55:55.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:55:55.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:55:55.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:55:55.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:55:55.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:55:55.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:55:55.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:55:55.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:55.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:55.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:55.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:55.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:55.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:55.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:55:55.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:55:55.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:55:55.236 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: expr 2 - 1 2026-02-20T16:55:55.237 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: seq 0 1 2026-02-20T16:55:55.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:55.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:55.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:55:55.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:55:55.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:55:55.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8be71c7b-0224-4dac-9cd0-227b1e1de8e8 2026-02-20T16:55:55.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8be71c7b-0224-4dac-9cd0-227b1e1de8e8' 2026-02-20T16:55:55.245 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 8be71c7b-0224-4dac-9cd0-227b1e1de8e8 2026-02-20T16:55:55.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:55:55.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbkphp44IdDxAAtHS/XpJWstuDSqXfEcxCOQ== 2026-02-20T16:55:55.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbkphp44IdDxAAtHS/XpJWstuDSqXfEcxCOQ=="}' 2026-02-20T16:55:55.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8be71c7b-0224-4dac-9cd0-227b1e1de8e8 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:55:55.345 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:55.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:55:55.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAbkphp44IdDxAAtHS/XpJWstuDSqXfEcxCOQ== --osd-uuid 8be71c7b-0224-4dac-9cd0-227b1e1de8e8 2026-02-20T16:55:55.372 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:55.369+0000 7f11682c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:55.374 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:55.369+0000 7f11682c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:55.375 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:55.373+0000 7f11682c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:55.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:55.373+0000 7f11682c6a40 -1 bdev(0x5590b0460c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:55:55.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:55.373+0000 7f11682c6a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:55:56.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:55:56.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:55:56.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:55:56.153 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:55:56.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:55:56.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:55:56.255 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:55:56.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:55:56.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:55:56.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:55:56.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:55:56.282 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:56.277+0000 7feeae286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:56.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:56.281+0000 7feeae286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:56.285 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:56.281+0000 7feeae286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:56.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:55:56.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:56.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:55:56.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:56.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:56.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:56.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:56.350 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:56.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:56.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:56.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:56.836 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:56.833+0000 7feeae286a40 -1 Falling back to public interface 2026-02-20T16:55:57.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:57.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:57.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:55:57.440 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:57.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:57.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:57.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:55:57.761 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:57.757+0000 7feeae286a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:55:58.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:55:58.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:58.533 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:55:58.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:55:58.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:58.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:55:58.690 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1832209722,v1:127.0.0.1:6801/1832209722] [v2:127.0.0.1:6802/1832209722,v1:127.0.0.1:6803/1832209722] exists,up 8be71c7b-0224-4dac-9cd0-227b1e1de8e8 2026-02-20T16:55:58.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:55:58.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:55:58.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:55:58.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:55:58.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:58.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:55:58.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:55:58.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:55:58.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b3672c7a-7af2-405b-9ef3-8d4d417a140b 2026-02-20T16:55:58.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b3672c7a-7af2-405b-9ef3-8d4d417a140b' 2026-02-20T16:55:58.699 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 b3672c7a-7af2-405b-9ef3-8d4d417a140b 2026-02-20T16:55:58.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:55:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAekphp73FHKhAAlr+B+Qk4/tWCro4rzG5CHw== 2026-02-20T16:55:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAekphp73FHKhAAlr+B+Qk4/tWCro4rzG5CHw=="}' 2026-02-20T16:55:58.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b3672c7a-7af2-405b-9ef3-8d4d417a140b -i td/osd-scrub-repair/1/new.json 2026-02-20T16:55:58.866 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:55:58.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:55:58.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAekphp73FHKhAAlr+B+Qk4/tWCro4rzG5CHw== --osd-uuid b3672c7a-7af2-405b-9ef3-8d4d417a140b 2026-02-20T16:55:58.893 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:58.889+0000 7f6286b55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:58.894 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:58.889+0000 7f6286b55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:58.895 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:58.893+0000 7f6286b55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:58.896 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:58.893+0000 7f6286b55a40 -1 bdev(0x562d8c246c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:55:58.896 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:58.893+0000 7f6286b55a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:55:59.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:55:59.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:55:59.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:55:59.611 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:55:59.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:55:59.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:55:59.820 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:55:59.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:55:59.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:55:59.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:55:59.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:55:59.836 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:59.833+0000 7f8ab3025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:59.838 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:59.833+0000 7f8ab3025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:59.839 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:55:59.837+0000 7f8ab3025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:55:59.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:55:59.986 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:55:59.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:55:59.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:55:59.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:00.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:00.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:00.277+0000 7f8ab3025a40 -1 Falling back to public interface 2026-02-20T16:56:00.922 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:00.917+0000 7f8ab3025a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:56:01.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:01.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:01.144 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:01.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:01.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:01.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:01.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:56:02.305 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:56:02.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:02.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:02.460 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/180155183,v1:127.0.0.1:6811/180155183] [v2:127.0.0.1:6812/180155183,v1:127.0.0.1:6813/180155183] exists,up b3672c7a-7af2-405b-9ef3-8d4d417a140b 2026-02-20T16:56:02.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:02.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:02.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:02.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats: create_pool testpool 1 1 2026-02-20T16:56:02.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-20T16:56:02.661 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:56:02.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:56:03.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats: ceph osd pool set testpool size 2 2026-02-20T16:56:03.874 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 size to 2 2026-02-20T16:56:03.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:731: TEST_repair_stats: wait_for_clean 2026-02-20T16:56:03.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:56:03.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:56:03.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:56:03.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:56:03.890 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:56:03.891 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:56:03.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:56:03.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:56:03.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:56:04.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:56:04.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:56:04.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:56:04.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:56:04.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:56:04.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:56:04.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:56:04.179 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:56:04.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:56:04.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:04.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:56:04.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:56:04.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:56:04.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:56:04.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:04.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:56:04.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-20T16:56:04.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-20T16:56:04.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-20T16:56:04.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:04.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:56:04.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:04.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:56:04.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:56:04.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:04.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:56:04.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:56:04.331 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:56:04.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:04.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T16:56:04.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:05.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:56:05.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:05.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T16:56:05.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:05.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-20T16:56:05.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:05.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:56:05.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-20T16:56:05.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:05.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-20T16:56:05.665 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705666 2026-02-20T16:56:05.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-20T16:56:05.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:56:05.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-20T16:56:05.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:56:05.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:05.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:06.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:56:06.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:56:06.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:56:06.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:56:06.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:56:06.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:06.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:06.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:56:06.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:56:06.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:56:06.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats: local payload=ABCDEF 2026-02-20T16:56:06.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats: echo ABCDEF 2026-02-20T16:56:06.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: seq 1 30 2026-02-20T16:56:06.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:06.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:07.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:07.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:07.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:07.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-20T16:56:07.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:07.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: get_not_primary testpool obj1 2026-02-20T16:56:07.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:56:07.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T16:56:07.137 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-20T16:56:07.137 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:56:07.137 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:56:07.137 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:07.137 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:56:07.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:56:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:56:07.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: local other=0 2026-02-20T16:56:07.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: get_pg testpool obj1 2026-02-20T16:56:07.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:56:07.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-20T16:56:07.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-20T16:56:07.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:56:07.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: local pgid=1.0 2026-02-20T16:56:07.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: get_primary testpool obj1 2026-02-20T16:56:07.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:56:07.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:56:07.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:07.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:56:07.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: local primary=1 2026-02-20T16:56:07.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:56:07.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:07.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:07.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:07.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:07.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:08.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:08.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:56:08.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:08.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:08.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:08.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:08.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:08.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:08.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: seq 1 20 2026-02-20T16:56:08.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:08.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 1 % 2 2026-02-20T16:56:08.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:08.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-20T16:56:08.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:08.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:08.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:08.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-20T16:56:08.595 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T16:56:08.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:08.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 2 % 2 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:08.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-20T16:56:08.881 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:104778fc:::obj2:head# 2026-02-20T16:56:08.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:08.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 3 % 2 2026-02-20T16:56:08.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-20T16:56:09.186 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:8dd16f86:::obj3:head# 2026-02-20T16:56:09.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:09.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 4 % 2 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-20T16:56:09.464 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:0ee9ae15:::obj4:head# 2026-02-20T16:56:09.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:09.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 5 % 2 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:09.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-20T16:56:09.779 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:head# 2026-02-20T16:56:09.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:09.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 6 % 2 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:09.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:09.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:09.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-20T16:56:10.051 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:233a42c1:::obj6:head# 2026-02-20T16:56:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:10.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 7 % 2 2026-02-20T16:56:10.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:10.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-20T16:56:10.370 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:9a09113a:::obj7:head# 2026-02-20T16:56:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:10.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 8 % 2 2026-02-20T16:56:10.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:10.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:10.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-20T16:56:10.659 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:a557efb1:::obj8:head# 2026-02-20T16:56:10.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:10.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 9 % 2 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-20T16:56:10.962 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:fe351e27:::obj9:head# 2026-02-20T16:56:11.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:11.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 10 % 2 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-20T16:56:11.222 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:d1337354:::obj10:head# 2026-02-20T16:56:11.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:11.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 11 % 2 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:11.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-20T16:56:11.510 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:eeae2a94:::obj11:head# 2026-02-20T16:56:11.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:11.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 12 % 2 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:11.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-20T16:56:11.811 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:af5f95cb:::obj12:head# 2026-02-20T16:56:11.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:11.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 13 % 2 2026-02-20T16:56:11.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:11.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-20T16:56:12.095 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:a61ad63e:::obj13:head# 2026-02-20T16:56:12.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:12.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 14 % 2 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-20T16:56:12.418 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:e0a44829:::obj14:head# 2026-02-20T16:56:12.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:12.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 15 % 2 2026-02-20T16:56:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-20T16:56:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:12.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:12.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:12.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-20T16:56:12.705 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ab946124:::obj15:head# 2026-02-20T16:56:12.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:12.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 16 % 2 2026-02-20T16:56:12.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:12.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:12.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-20T16:56:12.982 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:461f8b5e:::obj16:head# 2026-02-20T16:56:13.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:13.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 17 % 2 2026-02-20T16:56:13.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:13.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-20T16:56:13.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-20T16:56:13.290 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:24fc6e92:::obj17:head# 2026-02-20T16:56:13.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:13.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 18 % 2 2026-02-20T16:56:13.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:13.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-20T16:56:13.567 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:b9836a99:::obj18:head# 2026-02-20T16:56:13.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:13.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 19 % 2 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:13.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-20T16:56:13.857 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:00b31b6c:::obj19:head# 2026-02-20T16:56:13.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:13.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 20 % 2 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-20T16:56:14.122 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:24b5611b:::obj20:head# 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:56:14.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:14.216 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:14.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:56:14.218 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:56:14.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:56:14.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:14.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:56:14.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:56:14.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:56:14.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:56:14.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:56:14.236 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:14.232+0000 7f8ae5ba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:14.237 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:14.232+0000 7f8ae5ba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:14.238 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:14.232+0000 7f8ae5ba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:14.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:56:14.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:14.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:14.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:14.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:14.660 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:14.656+0000 7f8ae5ba2a40 -1 Falling back to public interface 2026-02-20T16:56:15.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:15.352+0000 7f8ae5ba2a40 -1 osd.1 17 log_to_monitors true 2026-02-20T16:56:15.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:15.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:15.541 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:15.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:15.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:15.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:15.703 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [9,17) [v2:127.0.0.1:6800/1193115304,v1:127.0.0.1:6801/1193115304] [v2:127.0.0.1:6802/1193115304,v1:127.0.0.1:6803/1193115304] exists,up b3672c7a-7af2-405b-9ef3-8d4d417a140b 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:15.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:15.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:15.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:15.705 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:15.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:15.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:56:15.707 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:56:15.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:56:15.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:15.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:56:15.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:56:15.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:56:15.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:56:15.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:56:15.725 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:15.720+0000 7f8276fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:15.726 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:15.720+0000 7f8276fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:15.727 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:15.724+0000 7f8276fdaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:15.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:16.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:16.156 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:16.152+0000 7f8276fdaa40 -1 Falling back to public interface 2026-02-20T16:56:16.785 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:16.780+0000 7f8276fdaa40 -1 osd.0 15 log_to_monitors true 2026-02-20T16:56:17.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:17.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:17.031 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:17.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:17.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:17.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:17.186 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 16 last_clean_interval [5,15) [v2:127.0.0.1:6810/978017362,v1:127.0.0.1:6811/978017362] [v2:127.0.0.1:6812/978017362,v1:127.0.0.1:6813/978017362] exists,up 8be71c7b-0224-4dac-9cd0-227b1e1de8e8 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats: wait_for_clean 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:56:17.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:56:17.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:56:17.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:56:17.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:56:17.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:56:17.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:56:17.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:56:17.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:56:17.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:56:17.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:56:17.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:56:17.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:56:17.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:56:17.473 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:56:17.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:56:17.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:17.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:56:17.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:56:17.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:56:17.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:56:17.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:17.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:56:17.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345922 2026-02-20T16:56:17.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345922 2026-02-20T16:56:17.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-85899345922' 2026-02-20T16:56:17.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:17.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:56:17.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:17.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:56:17.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:56:17.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:17.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:56:17.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:56:17.625 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 98784247810 2026-02-20T16:56:17.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:17.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:56:17.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:18.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:56:18.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:18.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-20T16:56:18.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:19.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:56:19.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:20.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-20T16:56:20.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:20.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345922 2026-02-20T16:56:20.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:20.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:56:20.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345922 2026-02-20T16:56:20.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:20.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345922 2026-02-20T16:56:20.113 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 85899345922 2026-02-20T16:56:20.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345922' 2026-02-20T16:56:20.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:56:20.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345922 2026-02-20T16:56:20.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:56:20.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:20.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:56:20.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:56:20.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:56:20.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:56:20.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:56:20.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:56:20.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:56:20.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:56:20.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:56:20.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:56:20.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:20.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:20.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:56:20.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats: repair 1.0 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:20.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:20.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:20.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T16:56:21.118 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-20T16:56:21.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:21.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:56:21.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:21.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:21.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:21.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:21.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:02.661637+0000 '>' 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:21.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:56:22.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:56:22.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:56:22.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:56:22.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:22.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:22.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:22.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:22.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:02.661637+0000 '>' 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:22.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:56:23.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:56:23.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:56:23.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:56:23.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:23.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:23.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:23.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:23.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:02.661637+0000 '>' 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:23.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:56:24.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:56:24.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:56:24.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:56:24.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:24.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:24.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:24.609 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:24.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:02.661637+0000 '>' 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:24.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:56:25.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:56:25.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:56:25.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:56:25.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:56:25.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:56:25.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:56:25.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:56:25.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:21.276795+0000 '>' 2026-02-20T16:56:02.661637+0000 2026-02-20T16:56:25.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:56:25.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats: wait_for_clean 2026-02-20T16:56:25.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:56:25.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:56:25.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:56:25.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:56:25.930 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:56:25.930 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:56:25.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:56:25.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:56:25.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:56:26.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:56:26.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:56:26.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:56:26.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:56:26.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:56:26.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:56:26.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:56:26.217 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:56:26.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:56:26.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:26.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:56:26.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247812 2026-02-20T16:56:26.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247812 2026-02-20T16:56:26.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812' 2026-02-20T16:56:26.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:26.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:56:26.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345925 2026-02-20T16:56:26.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345925 2026-02-20T16:56:26.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2026-02-20T16:56:26.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:26.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247812 2026-02-20T16:56:26.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:26.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:56:26.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247812 2026-02-20T16:56:26.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:26.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247812 2026-02-20T16:56:26.367 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 98784247812 2026-02-20T16:56:26.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2026-02-20T16:56:26.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:26.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247812 2026-02-20T16:56:26.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:27.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:56:27.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:27.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247812 2026-02-20T16:56:27.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:27.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345925 2026-02-20T16:56:27.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:56:27.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345925 2026-02-20T16:56:27.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:27.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345925 2026-02-20T16:56:27.688 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 85899345925 2026-02-20T16:56:27.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2026-02-20T16:56:27.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:56:27.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345925 -lt 85899345925 2026-02-20T16:56:27.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:56:27.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:27.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:56:28.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:56:28.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:56:28.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:56:28.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:56:28.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:56:28.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:56:28.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:56:28.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:56:28.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:56:28.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:28.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:28.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:56:28.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:56:28.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:56:28.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats: ceph pg dump pgs 2026-02-20T16:56:28.549 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:56:28.549 INFO:tasks.workunit.client.0.trial013.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-20T16:56:21.293878+0000 15'30 25:132 [1,0] 1 [1,0] 1 15'30 2026-02-20T16:56:21.276795+0000 15'30 2026-02-20T16:56:21.276795+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:56:21.276795+0000 20 0 2026-02-20T16:56:28.549 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:56:28.549 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:56:28.550 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:56:28.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:762: TEST_repair_stats: flush_pg_stats 2026-02-20T16:56:28.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:56:28.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:56:28.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:56:28.732 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:56:28.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:56:28.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:28.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:56:28.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247814 2026-02-20T16:56:28.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247814 2026-02-20T16:56:28.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247814' 2026-02-20T16:56:28.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:28.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:56:28.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345926 2026-02-20T16:56:28.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345926 2026-02-20T16:56:28.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247814 1-85899345926' 2026-02-20T16:56:28.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:28.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247814 2026-02-20T16:56:28.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:28.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:56:28.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247814 2026-02-20T16:56:28.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:28.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247814 2026-02-20T16:56:28.883 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 98784247814 2026-02-20T16:56:28.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2026-02-20T16:56:28.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:29.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247814 2026-02-20T16:56:29.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:30.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:56:30.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:30.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-20T16:56:30.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:30.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345926 2026-02-20T16:56:30.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:30.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:56:30.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345926 2026-02-20T16:56:30.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:30.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345926 2026-02-20T16:56:30.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345926' 2026-02-20T16:56:30.210 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 85899345926 2026-02-20T16:56:30.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:56:30.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345926 -lt 85899345926 2026-02-20T16:56:30.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: ceph pg 1.0 query 2026-02-20T16:56:30.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: jq .info.stats.stat_sum 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes": 210, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects": 30, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_object_clones": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_object_copies": 60, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_missing_on_primary": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_missing": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_degraded": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_misplaced": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_unfound": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_dirty": 30, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_whiteouts": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_read": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_read_kb": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_write": 30, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_write_kb": 30, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_scrub_errors": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_shallow_scrub_errors": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_deep_scrub_errors": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_recovered": 20, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes_recovered": 140, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_keys_recovered": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_omap": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_hit_set_archive": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes_hit_set_archive": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_kb": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_kb": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_promote": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_mode_high": 0, 2026-02-20T16:56:30.458 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_mode_low": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_mode_some": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_mode_full": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_pinned": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_legacy_snapsets": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_large_omap_objects": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_manifest": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_omap_bytes": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_omap_keys": 0, 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_repaired": 20 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: ceph pg 1.0 query 2026-02-20T16:56:30.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2026-02-20T16:56:30.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: COUNT=20 2026-02-20T16:56:30.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats: test 20 = 20 2026-02-20T16:56:30.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:30.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-20T16:56:30.675 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 20, 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 85899345926, 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 1, 2026-02-20T16:56:30.684 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 1, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 1, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 1, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 104857600, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 28568, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 328, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 7, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 28216, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 104829032, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "total": 107374182400, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "available": 107344928768, 2026-02-20T16:56:30.685 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 335872, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 56545, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 7655, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 28893721 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [ 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: 0 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:56:30.686 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 10, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:30.687 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [] 2026-02-20T16:56:30.688 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:56:30.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:30.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-20T16:56:30.834 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:30.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: COUNT=10 2026-02-20T16:56:30.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: expr 20 / 2 2026-02-20T16:56:30.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: test 10 = 10 2026-02-20T16:56:30.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:30.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-20T16:56:30.991 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:30.999 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:56:30.999 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 23, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 98784247814, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 1, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 1, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 1, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 1, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 104857600, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 28568, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 328, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 6, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 28217, 2026-02-20T16:56:31.000 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 104829032, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "total": 107374182400, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "available": 107344928768, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 335872, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 56545, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 6354, 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 28895022 2026-02-20T16:56:31.001 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [ 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 10, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:56:31.002 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [] 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:56:31.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:31.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-20T16:56:31.150 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:31.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: COUNT=10 2026-02-20T16:56:31.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: expr 20 / 2 2026-02-20T16:56:31.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: test 10 = 10 2026-02-20T16:56:31.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:31.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: jq .pg_map.osd_stats_sum 2026-02-20T16:56:31.313 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:31.323 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:56:31.323 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 0, 2026-02-20T16:56:31.323 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 2, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 2, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 2, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 2, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 209715200, 2026-02-20T16:56:31.324 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 57136, 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 656, 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 13, 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 56434, 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 209658064, 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:56:31.325 INFO:tasks.workunit.client.0.trial013.stdout: "total": 214748364800, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "available": 214689857536, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 671744, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 113090, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:56:31.326 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 14009, 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 57788743 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [], 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:56:31.327 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 20, 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:56:31.328 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [], 2026-02-20T16:56:31.329 INFO:tasks.workunit.client.0.trial013.stdout: "network_ping_times": [] 2026-02-20T16:56:31.330 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:56:31.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-20T16:56:31.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-20T16:56:31.472 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: COUNT=20 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats: test 20 = 20 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:56:31.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:56:31.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:31.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:31.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:31.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:31.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:31.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:31.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:56:31.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:56:31.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:56:31.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:56:31.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:56:31.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:56:31.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:56:31.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:56:31.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:56:31.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:56:31.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:56:31.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:56:31.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:56:31.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:56:31.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:56:31.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:56:31.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:31.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:31.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:31.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:56:31.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:56:31.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:56:31.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:56:31.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:56:31.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:56:31.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:56:31.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:56:31.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:56:31.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:56:31.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:56:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:56:31.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:56:31.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:56:31.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:56:31.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:56:31.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:56:31.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:56:31.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:56:31.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:56:31.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: local poolname=testpool 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: local OSDS=3 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: local OBJS=30 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: local REPAIRS=26 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: local allow_overwrites=false 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2026-02-20T16:56:31.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:56:31.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:56:31.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:56:31.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:56:31.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:31.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:56:31.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:56:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:56:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:56:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:56:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:56:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.688 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.689 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:56:31.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:56:31.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:56:31.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:56:31.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:56:31.745 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:56:31.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:56:31.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.746 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.746 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:56:31.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:56:31.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:56:31.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:56:31.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:56:31.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:31.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:31.895 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:31.895 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.896 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:31.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:56:31.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:56:31.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:31.917 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: expr 3 - 1 2026-02-20T16:56:31.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: seq 0 2 2026-02-20T16:56:31.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:56:31.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:31.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:31.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:56:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:56:31.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:56:31.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:56:31.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=46d23770-e6ad-45ce-a008-870331683f16 2026-02-20T16:56:31.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 46d23770-e6ad-45ce-a008-870331683f16' 2026-02-20T16:56:31.926 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 46d23770-e6ad-45ce-a008-870331683f16 2026-02-20T16:56:31.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:56:31.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA/kphp1kPQNxAAxtIJKNHk528K8fGx4dVosA== 2026-02-20T16:56:31.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA/kphp1kPQNxAAxtIJKNHk528K8fGx4dVosA=="}' 2026-02-20T16:56:31.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 46d23770-e6ad-45ce-a008-870331683f16 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:56:32.029 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:32.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:56:32.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA/kphp1kPQNxAAxtIJKNHk528K8fGx4dVosA== --osd-uuid 46d23770-e6ad-45ce-a008-870331683f16 2026-02-20T16:56:32.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.048+0000 7fc64b871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.056 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.052+0000 7fc64b871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.057 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.052+0000 7fc64b871a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.058 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.052+0000 7fc64b871a40 -1 bdev(0x55e5cdd8ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:56:32.058 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.052+0000 7fc64b871a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:56:32.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:56:32.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:56:32.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:56:32.792 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:56:32.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:56:32.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:56:32.895 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:56:32.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:32.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:56:32.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:56:32.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:56:32.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.916+0000 7fa9fe550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.923 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.920+0000 7fa9fe550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.925 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:32.920+0000 7fa9fe550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:32.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:56:32.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:32.988 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:32.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:32.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:33.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:33.341 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:33.336+0000 7fa9fe550a40 -1 Falling back to public interface 2026-02-20T16:56:34.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:34.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:34.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:34.082 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:34.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:34.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:34.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:34.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:34.352+0000 7fa9fe550a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:56:35.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:35.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:35.196 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:56:35.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:56:35.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:35.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:35.358 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4203624661,v1:127.0.0.1:6801/4203624661] [v2:127.0.0.1:6802/4203624661,v1:127.0.0.1:6803/4203624661] exists,up 46d23770-e6ad-45ce-a008-870331683f16 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:56:35.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:35.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:35.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:35.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:56:35.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:56:35.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2e22eaf6-65f8-4665-8f2e-b39f412f1a60 2026-02-20T16:56:35.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2e22eaf6-65f8-4665-8f2e-b39f412f1a60' 2026-02-20T16:56:35.366 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 2e22eaf6-65f8-4665-8f2e-b39f412f1a60 2026-02-20T16:56:35.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:56:35.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBDkphpdQByFhAAK8qFlhFs75Q7bYubp87IJw== 2026-02-20T16:56:35.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBDkphpdQByFhAAK8qFlhFs75Q7bYubp87IJw=="}' 2026-02-20T16:56:35.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2e22eaf6-65f8-4665-8f2e-b39f412f1a60 -i td/osd-scrub-repair/1/new.json 2026-02-20T16:56:35.535 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:35.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:56:35.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBDkphpdQByFhAAK8qFlhFs75Q7bYubp87IJw== --osd-uuid 2e22eaf6-65f8-4665-8f2e-b39f412f1a60 2026-02-20T16:56:35.562 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:35.556+0000 7f6f01feda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:35.563 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:35.560+0000 7f6f01feda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:35.565 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:35.560+0000 7f6f01feda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:35.565 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:35.560+0000 7f6f01feda40 -1 bdev(0x561ea414cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:56:35.565 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:35.560+0000 7f6f01feda40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:56:36.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:56:36.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:56:36.231 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:56:36.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:56:36.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:56:36.436 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:56:36.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:56:36.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:36.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:56:36.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:56:36.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:56:36.453 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:36.448+0000 7f9919e51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:36.455 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:36.448+0000 7f9919e51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:36.456 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:36.452+0000 7f9919e51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:36.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:36.600 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:36.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:36.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:36.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:36.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:36.857 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:36.852+0000 7f9919e51a40 -1 Falling back to public interface 2026-02-20T16:56:37.529 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:37.524+0000 7f9919e51a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:56:37.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:37.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:37.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:37.763 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:37.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:37.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:37.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:38.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:38.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:38.922 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:56:38.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:56:38.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:38.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:39.085 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3849879362,v1:127.0.0.1:6811/3849879362] [v2:127.0.0.1:6812/3849879362,v1:127.0.0.1:6813/3849879362] exists,up 2e22eaf6-65f8-4665-8f2e-b39f412f1a60 2026-02-20T16:56:39.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:39.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:39.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:39.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:39.091 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:39.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:39.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:56:39.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:39.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:56:39.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:56:39.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17cdb6bf-9a2d-425e-af47-a1371139f74e 2026-02-20T16:56:39.094 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 17cdb6bf-9a2d-425e-af47-a1371139f74e 2026-02-20T16:56:39.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 17cdb6bf-9a2d-425e-af47-a1371139f74e' 2026-02-20T16:56:39.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:56:39.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHkphpuMkYBhAAd8FLJf4bF+LWxdplIFtmxA== 2026-02-20T16:56:39.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHkphpuMkYBhAAd8FLJf4bF+LWxdplIFtmxA=="}' 2026-02-20T16:56:39.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17cdb6bf-9a2d-425e-af47-a1371139f74e -i td/osd-scrub-repair/2/new.json 2026-02-20T16:56:39.265 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:56:39.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:56:39.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBHkphpuMkYBhAAd8FLJf4bF+LWxdplIFtmxA== --osd-uuid 17cdb6bf-9a2d-425e-af47-a1371139f74e 2026-02-20T16:56:39.292 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:39.288+0000 7fe2da0e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:39.294 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:39.288+0000 7fe2da0e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:39.296 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:39.288+0000 7fe2da0e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:39.296 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:39.292+0000 7fe2da0e1a40 -1 bdev(0x55e98c9f0c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:56:39.296 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:39.292+0000 7fe2da0e1a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:56:39.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:56:39.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:56:39.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:56:39.968 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:56:39.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:56:40.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:56:40.186 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:40.180+0000 7f1127b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:40.188 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:40.184+0000 7f1127b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:40.189 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:40.184+0000 7f1127b3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:40.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:56:40.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:40.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:56:40.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:40.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:40.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:40.319 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:40.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:40.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:40.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:56:40.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:40.645 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:40.640+0000 7f1127b3ca40 -1 Falling back to public interface 2026-02-20T16:56:41.274 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:41.268+0000 7f1127b3ca40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:56:41.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:41.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:41.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:41.471 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:41.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:41.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:56:41.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:42.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:42.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:42.632 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:56:42.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:56:42.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:42.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:56:42.787 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4058932321,v1:127.0.0.1:6819/4058932321] [v2:127.0.0.1:6820/4058932321,v1:127.0.0.1:6821/4058932321] exists,up 17cdb6bf-9a2d-425e-af47-a1371139f74e 2026-02-20T16:56:42.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:42.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:42.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=testpool 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T16:56:42.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-20T16:56:43.031 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T16:56:43.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2026-02-20T16:56:43.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2026-02-20T16:56:43.336 INFO:tasks.workunit.client.0.trial013.stderr:pool 'testpool' created 2026-02-20T16:56:43.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:56:44.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T16:56:44.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T16:56:44.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:56:44.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:56:44.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:56:44.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:56:44.349 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:56:44.349 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:56:44.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:56:44.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:56:44.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:56:44.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:56:44.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:56:44.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:56:44.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:56:44.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:56:44.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:56:44.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:56:44.628 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:56:44.628 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:56:44.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:56:44.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:44.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:56:44.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:56:44.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:56:44.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:56:44.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:44.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:56:44.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T16:56:44.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T16:56:44.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-20T16:56:44.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:56:44.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:56:44.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509442 2026-02-20T16:56:44.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509442 2026-02-20T16:56:44.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-64424509442' 2026-02-20T16:56:44.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:44.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:56:44.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:44.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:56:44.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:56:44.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:44.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:56:44.851 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:56:44.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:56:44.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:45.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:56:45.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:46.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:56:46.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:46.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:56:46.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:56:47.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:56:47.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:56:47.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:56:47.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:47.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T16:56:47.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:47.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:56:47.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T16:56:47.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:47.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T16:56:47.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T16:56:47.351 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T16:56:47.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:56:47.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T16:56:47.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:56:47.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509442 2026-02-20T16:56:47.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:56:47.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:56:47.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509442 2026-02-20T16:56:47.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:56:47.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509442 2026-02-20T16:56:47.528 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509442 2026-02-20T16:56:47.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2026-02-20T16:56:47.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:56:47.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509442 2026-02-20T16:56:47.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:56:47.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:47.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:56:47.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:56:47.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:56:47.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:56:48.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:56:48.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:56:48.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:56:48.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:56:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:56:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:56:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:56:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T16:56:48.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: TEST_repair_stats_ec: local payload=ABCDEF 2026-02-20T16:56:48.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: TEST_repair_stats_ec: echo ABCDEF 2026-02-20T16:56:48.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: seq 1 30 2026-02-20T16:56:48.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-20T16:56:48.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-20T16:56:48.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: get_not_primary testpool obj1 2026-02-20T16:56:48.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-20T16:56:48.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T16:56:48.974 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-20T16:56:48.974 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:56:48.974 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:56:48.974 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:48.974 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:56:49.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:56:49.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:49.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:56:49.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: local other=0 2026-02-20T16:56:49.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: get_pg testpool obj1 2026-02-20T16:56:49.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-20T16:56:49.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-20T16:56:49.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-20T16:56:49.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:56:49.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: local pgid=1.0 2026-02-20T16:56:49.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: get_primary testpool obj1 2026-02-20T16:56:49.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-20T16:56:49.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T16:56:49.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-20T16:56:49.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:56:49.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: local primary=1 2026-02-20T16:56:49.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:817: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:56:49.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:49.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:49.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:49.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:49.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:49.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:49.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:818: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-20T16:56:49.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:56:49.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:56:49.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:56:49.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:56:49.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:56:50.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:56:50.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: seq 1 26 2026-02-20T16:56:50.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:50.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 1 % 2 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:50.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-20T16:56:50.459 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:ff7b1f36:::obj1:head# 2026-02-20T16:56:50.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:50.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 2 % 2 2026-02-20T16:56:50.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:50.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-20T16:56:50.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:50.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:50.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:50.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-20T16:56:50.780 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:104778fc:::obj2:head# 2026-02-20T16:56:50.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:50.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 3 % 2 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:50.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:50.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-20T16:56:51.069 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:8dd16f86:::obj3:head# 2026-02-20T16:56:51.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:51.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 4 % 2 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-20T16:56:51.329 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:0ee9ae15:::obj4:head# 2026-02-20T16:56:51.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:51.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 5 % 2 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:51.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-20T16:56:51.622 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:c52c9666:::obj5:head# 2026-02-20T16:56:51.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:51.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 6 % 2 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:51.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:51.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:51.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-20T16:56:51.914 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:233a42c1:::obj6:head# 2026-02-20T16:56:52.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:52.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 7 % 2 2026-02-20T16:56:52.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:52.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-20T16:56:52.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:52.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:52.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:52.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-20T16:56:52.221 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:9a09113a:::obj7:head# 2026-02-20T16:56:52.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:52.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 8 % 2 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:52.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-20T16:56:52.502 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:a557efb1:::obj8:head# 2026-02-20T16:56:52.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:52.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 9 % 2 2026-02-20T16:56:52.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:52.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-20T16:56:52.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:52.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:52.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:52.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-20T16:56:52.826 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:fe351e27:::obj9:head# 2026-02-20T16:56:52.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:52.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 10 % 2 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-20T16:56:53.123 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:d1337354:::obj10:head# 2026-02-20T16:56:53.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:53.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 11 % 2 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:53.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-20T16:56:53.448 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:eeae2a94:::obj11:head# 2026-02-20T16:56:53.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:53.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 12 % 2 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:53.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-20T16:56:53.746 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:af5f95cb:::obj12:head# 2026-02-20T16:56:53.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:53.856 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 13 % 2 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:53.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:53.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:53.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-20T16:56:54.019 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:a61ad63e:::obj13:head# 2026-02-20T16:56:54.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:54.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 14 % 2 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:54.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-20T16:56:54.307 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:e0a44829:::obj14:head# 2026-02-20T16:56:54.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:54.404 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 15 % 2 2026-02-20T16:56:54.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:54.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-20T16:56:54.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:54.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:54.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:54.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-20T16:56:54.567 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:ab946124:::obj15:head# 2026-02-20T16:56:54.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:54.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 16 % 2 2026-02-20T16:56:54.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:54.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:54.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-20T16:56:54.857 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:461f8b5e:::obj16:head# 2026-02-20T16:56:54.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:54.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 17 % 2 2026-02-20T16:56:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-20T16:56:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:54.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:54.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:54.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-20T16:56:55.115 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:24fc6e92:::obj17:head# 2026-02-20T16:56:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:55.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 18 % 2 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:55.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-20T16:56:55.415 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:b9836a99:::obj18:head# 2026-02-20T16:56:55.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:55.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 19 % 2 2026-02-20T16:56:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:55.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:55.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-20T16:56:55.674 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:00b31b6c:::obj19:head# 2026-02-20T16:56:55.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:55.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 20 % 2 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:55.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:55.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:55.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-20T16:56:55.983 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:24b5611b:::obj20:head# 2026-02-20T16:56:56.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:56.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 21 % 2 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:56.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2026-02-20T16:56:56.247 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:1243fe1a:::obj21:head# 2026-02-20T16:56:56.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:56.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 22 % 2 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:56.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2026-02-20T16:56:56.555 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:c0c49def:::obj22:head# 2026-02-20T16:56:56.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:56.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 23 % 2 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:56.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2026-02-20T16:56:56.834 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:242c40e0:::obj23:head# 2026-02-20T16:56:56.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:56.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 24 % 2 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:56.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2026-02-20T16:56:57.125 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:7eacede5:::obj24:head# 2026-02-20T16:56:57.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:57.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 25 % 2 2026-02-20T16:56:57.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-20T16:56:57.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2026-02-20T16:56:57.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:57.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T16:56:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2026-02-20T16:56:57.395 INFO:tasks.workunit.client.0.trial013.stdout:remove 0#1:7109478f:::obj25:head# 2026-02-20T16:56:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-20T16:56:57.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 26 % 2 2026-02-20T16:56:57.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-20T16:56:57.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2026-02-20T16:56:57.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:56:57.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:56:57.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:56:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:56:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:57.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2026-02-20T16:56:57.690 INFO:tasks.workunit.client.0.trial013.stdout:remove 1#1:f3c2593a:::obj26:head# 2026-02-20T16:56:57.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:57.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:57.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:56:57.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T16:56:57.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:57.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:57.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:57.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:56:57.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:57.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:56:57.802 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:56:57.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:56:57.817 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:57.815+0000 7f2d92c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:57.819 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:57.815+0000 7f2d92c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:57.820 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:57.819+0000 7f2d92c07a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:57.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T16:56:57.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:57.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:56:57.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:57.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:57.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:57.968 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:57.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:57.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:57.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:58.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:58.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:58.279+0000 7f2d92c07a40 -1 Falling back to public interface 2026-02-20T16:56:58.951 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:58.947+0000 7f2d92c07a40 -1 osd.1 23 log_to_monitors true 2026-02-20T16:56:59.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:56:59.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:59.130 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:56:59.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:56:59.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:59.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:56:59.289 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6800/759293789,v1:127.0.0.1:6801/759293789] [v2:127.0.0.1:6802/759293789,v1:127.0.0.1:6803/759293789] exists,up 2e22eaf6-65f8-4665-8f2e-b39f412f1a60 2026-02-20T16:56:59.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:56:59.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:56:59.294 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:56:59.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-20T16:56:59.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:56:59.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:56:59.296 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:56:59.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:56:59.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:59.315+0000 7f61e5f69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:59.318 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:59.315+0000 7f61e5f69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:59.319 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:59.319+0000 7f61e5f69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:56:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:56:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:56:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:56:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:56:59.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:56:59.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:56:59.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:56:59.453 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:56:59.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:56:59.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:56:59.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:56:59.717 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:56:59.715+0000 7f61e5f69a40 -1 Falling back to public interface 2026-02-20T16:57:00.371 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:00.367+0000 7f61e5f69a40 -1 osd.0 22 log_to_monitors true 2026-02-20T16:57:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:57:00.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:57:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:57:00.615 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:57:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:57:00.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:57:00.772 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,22) [v2:127.0.0.1:6810/741647137,v1:127.0.0.1:6811/741647137] [v2:127.0.0.1:6812/741647137,v1:127.0.0.1:6813/741647137] exists,up 46d23770-e6ad-45ce-a008-870331683f16 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: TEST_repair_stats_ec: wait_for_clean 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:57:00.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:57:00.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:57:00.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:57:00.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:57:00.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:57:00.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:57:00.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:57:00.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:57:01.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:57:01.053 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:57:01.054 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:57:01.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:57:01.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:01.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:57:01.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-20T16:57:01.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-20T16:57:01.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-20T16:57:01.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:01.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:57:01.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116994 2026-02-20T16:57:01.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116994 2026-02-20T16:57:01.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-115964116994' 2026-02-20T16:57:01.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:01.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:57:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509446 2026-02-20T16:57:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509446 2026-02-20T16:57:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-115964116994 2-64424509446' 2026-02-20T16:57:01.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:01.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-20T16:57:01.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:01.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:57:01.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-20T16:57:01.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:01.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-20T16:57:01.279 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 128849018882 2026-02-20T16:57:01.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-20T16:57:01.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:01.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 128849018882 2026-02-20T16:57:01.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:02.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:57:02.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:02.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-20T16:57:02.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:02.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116994 2026-02-20T16:57:02.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:02.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:57:02.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116994 2026-02-20T16:57:02.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:02.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116994 2026-02-20T16:57:02.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116994' 2026-02-20T16:57:02.611 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 115964116994 2026-02-20T16:57:02.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:57:02.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116994 -lt 115964116994 2026-02-20T16:57:02.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:02.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509446 2026-02-20T16:57:02.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:02.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:57:02.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509446 2026-02-20T16:57:02.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:02.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509446 2026-02-20T16:57:02.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-20T16:57:02.787 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509446 2026-02-20T16:57:02.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:57:02.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509446 -lt 64424509446 2026-02-20T16:57:02.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:57:02.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:02.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:03.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:57:03.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:57:03.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:57:03.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:57:03.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:03.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:03.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:57:03.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:57:03.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: TEST_repair_stats_ec: repair 1.0 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:03.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:03.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T16:56:43.334281+0000 2026-02-20T16:57:03.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T16:57:03.819 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-20T16:57:03.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T16:56:43.334281+0000 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:56:43.334281+0000 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:03.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:03.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:03.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:03.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:56:43.334281+0000 '>' 2026-02-20T16:56:43.334281+0000 2026-02-20T16:57:03.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:04.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:04.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:04.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:04.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:04.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:04.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:05.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:57:04.002943+0000 '>' 2026-02-20T16:56:43.334281+0000 2026-02-20T16:57:05.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:57:05.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: TEST_repair_stats_ec: wait_for_clean 2026-02-20T16:57:05.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:57:05.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:57:05.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:57:05.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:57:05.155 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:57:05.155 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:57:05.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:57:05.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:57:05.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:57:05.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:57:05.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:57:05.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:57:05.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:57:05.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:57:05.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:57:05.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:57:05.434 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:57:05.435 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:57:05.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:57:05.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:05.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:57:05.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-20T16:57:05.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-20T16:57:05.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884' 2026-02-20T16:57:05.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:05.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:57:05.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-20T16:57:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-20T16:57:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-115964116996' 2026-02-20T16:57:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:05.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:57:05.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509448 2026-02-20T16:57:05.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509448 2026-02-20T16:57:05.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-115964116996 2-64424509448' 2026-02-20T16:57:05.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:05.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018884 2026-02-20T16:57:05.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:05.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:57:05.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018884 2026-02-20T16:57:05.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:05.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-20T16:57:05.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2026-02-20T16:57:05.656 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 128849018884 2026-02-20T16:57:05.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:05.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018884 2026-02-20T16:57:05.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:06.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:57:06.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:06.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-20T16:57:06.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:06.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116996 2026-02-20T16:57:06.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:06.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:57:06.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116996 2026-02-20T16:57:06.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:06.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116996 2026-02-20T16:57:06.986 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 115964116996 2026-02-20T16:57:06.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116996' 2026-02-20T16:57:06.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:57:07.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-20T16:57:07.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:07.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509448 2026-02-20T16:57:07.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:07.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:57:07.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509448 2026-02-20T16:57:07.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:07.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509448 2026-02-20T16:57:07.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2026-02-20T16:57:07.163 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509448 2026-02-20T16:57:07.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:57:07.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509448 -lt 64424509448 2026-02-20T16:57:07.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:57:07.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:07.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:07.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:57:07.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:57:07.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:57:07.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:57:07.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:57:07.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:07.671 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:07.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:57:07.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:57:07.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:57:07.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: TEST_repair_stats_ec: ceph pg dump pgs 2026-02-20T16:57:08.010 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T16:57:08.010 INFO:tasks.workunit.client.0.trial013.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-20T16:57:04.037703+0000 21'30 31:137 [1,0,2] 1 [1,0,2] 1 21'30 2026-02-20T16:57:04.002943+0000 21'30 2026-02-20T16:57:04.002943+0000 0 1 periodic scrub scheduled @ 2026-02-21T16:57:04.002943+0000 17 0 2026-02-20T16:57:08.010 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T16:57:08.010 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T16:57:08.010 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T16:57:08.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: TEST_repair_stats_ec: flush_pg_stats 2026-02-20T16:57:08.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:57:08.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:57:08.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:57:08.193 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:57:08.194 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T16:57:08.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:57:08.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:08.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:57:08.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018885 2026-02-20T16:57:08.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018885 2026-02-20T16:57:08.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885' 2026-02-20T16:57:08.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:08.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:57:08.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-20T16:57:08.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-20T16:57:08.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885 1-115964116997' 2026-02-20T16:57:08.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:08.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T16:57:08.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509450 2026-02-20T16:57:08.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509450 2026-02-20T16:57:08.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885 1-115964116997 2-64424509450' 2026-02-20T16:57:08.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:08.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018885 2026-02-20T16:57:08.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:08.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:57:08.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018885 2026-02-20T16:57:08.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:08.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018885 2026-02-20T16:57:08.415 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 128849018885 2026-02-20T16:57:08.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2026-02-20T16:57:08.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:08.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018885 2026-02-20T16:57:08.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:09.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:57:09.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:09.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018885 2026-02-20T16:57:09.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:10.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:57:10.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:10.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018885 -lt 128849018885 2026-02-20T16:57:10.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:10.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116997 2026-02-20T16:57:10.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:10.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:57:10.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116997 2026-02-20T16:57:10.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:10.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116997 2026-02-20T16:57:10.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116997' 2026-02-20T16:57:10.913 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 115964116997 2026-02-20T16:57:10.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:57:11.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116998 -lt 115964116997 2026-02-20T16:57:11.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:11.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509450 2026-02-20T16:57:11.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:11.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T16:57:11.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509450 2026-02-20T16:57:11.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:11.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509450 2026-02-20T16:57:11.075 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509450 2026-02-20T16:57:11.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2026-02-20T16:57:11.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T16:57:11.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509450 2026-02-20T16:57:11.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-20T16:57:11.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: jq .info.stats.stat_sum 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes": 210, 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects": 30, 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_object_clones": 0, 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_object_copies": 90, 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_missing_on_primary": 0, 2026-02-20T16:57:11.320 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_missing": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_degraded": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_misplaced": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_unfound": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_dirty": 30, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_whiteouts": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_read": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_read_kb": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_write": 30, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_write_kb": 30, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_scrub_errors": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_shallow_scrub_errors": 0, 2026-02-20T16:57:11.321 INFO:tasks.workunit.client.0.trial013.stdout: "num_deep_scrub_errors": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_recovered": 26, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes_recovered": 182, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_keys_recovered": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_omap": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_hit_set_archive": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_bytes_hit_set_archive": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_kb": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_kb": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_promote": 0, 2026-02-20T16:57:11.322 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_mode_high": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_flush_mode_low": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_mode_some": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_evict_mode_full": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_pinned": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_legacy_snapsets": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_large_omap_objects": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_manifest": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_omap_bytes": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_omap_keys": 0, 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout: "num_objects_repaired": 26 2026-02-20T16:57:11.323 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:11.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-20T16:57:11.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2026-02-20T16:57:11.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: COUNT=26 2026-02-20T16:57:11.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: TEST_repair_stats_ec: test 26 = 26 2026-02-20T16:57:11.397 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: expr 3 - 1 2026-02-20T16:57:11.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: seq 0 2 2026-02-20T16:57:11.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:57:11.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2026-02-20T16:57:11.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-20T16:57:11.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-20T16:57:11.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:11.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-20T16:57:11.552 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:11.561 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:11.561 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T16:57:11.561 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 30, 2026-02-20T16:57:11.561 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 128849018885, 2026-02-20T16:57:11.561 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 1, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 1, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 1, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 1, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 104857600, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 28488, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 376, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 5, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 28090, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 104829112, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "total": 107374182400, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "available": 107345010688, 2026-02-20T16:57:11.562 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 385024, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 205045, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 5703, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 28764601 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [ 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: 1, 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: 2 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:57:11.563 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 13, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.564 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [] 2026-02-20T16:57:11.565 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:11.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:11.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-20T16:57:11.712 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:11.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-20T16:57:11.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-20T16:57:11.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:57:11.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2026-02-20T16:57:11.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-20T16:57:11.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-20T16:57:11.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:11.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-20T16:57:11.872 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:11.882 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 27, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 115964116998, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 1, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 1, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 1, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 1, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 104857600, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 28488, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 376, 2026-02-20T16:57:11.883 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 6, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 28089, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 104829112, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "total": 107374182400, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "available": 107345010688, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 385024, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 205045, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:57:11.884 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 7003, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 28763301 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [ 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: 0, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: 2 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 13, 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:57:11.885 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [] 2026-02-20T16:57:11.886 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:11.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:11.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-20T16:57:12.033 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: TEST_repair_stats_ec: repair=0 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:12.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2026-02-20T16:57:12.194 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 2, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 15, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 64424509450, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 1, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 1, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 1, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 1, 2026-02-20T16:57:12.204 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 104857600, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 27336, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 376, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 13, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 26930, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 104830264, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "total": 107374182400, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "available": 107346190336, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 385024, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 205045, 2026-02-20T16:57:12.205 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 13343, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 27577313 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [ 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:57:12.206 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 0, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.207 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [] 2026-02-20T16:57:12.208 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:12.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:12.208 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2026-02-20T16:57:12.354 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:12.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=0 2026-02-20T16:57:12.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 0 = 0 2026-02-20T16:57:12.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:12.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2026-02-20T16:57:12.512 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:12.521 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "up_from": 0, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "num_pgs": 3, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "num_osds": 3, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_osds": 3, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "num_per_pool_omap_osds": 3, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "kb": 314572800, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used": 84312, 2026-02-20T16:57:12.522 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_data": 1128, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_omap": 25, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "kb_used_meta": 83110, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "kb_avail": 314488488, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "statfs": { 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "total": 322122547200, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "available": 322036211712, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "internally_reserved": 0, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "allocated": 1155072, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "data_stored": 615135, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed": 0, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_allocated": 0, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "data_compressed_original": 0, 2026-02-20T16:57:12.523 INFO:tasks.workunit.client.0.trial013.stdout: "omap_allocated": 26049, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "internal_metadata": 85105215 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "hb_peers": [], 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "snap_trim_queue_len": 0, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "num_snap_trimming": 0, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "num_shards_repaired": 26, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "op_queue_age_hist": { 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "histogram": [], 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "upper_bound": 1 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.524 INFO:tasks.workunit.client.0.trial013.stdout: "perf_stat": { 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ms": 0, 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ms": 0, 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "commit_latency_ns": 0, 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "apply_latency_ns": 0 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "alerts": [], 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout: "network_ping_times": [] 2026-02-20T16:57:12.525 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:12.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-20T16:57:12.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-20T16:57:12.673 INFO:tasks.workunit.client.0.trial013.stderr:dumped all 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: COUNT=26 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: TEST_repair_stats_ec: test 26 = 26 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:57:12.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:57:12.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:57:12.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:57:12.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:57:12.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:57:12.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:57:12.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:57:12.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:57:12.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:57:12.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:57:12.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:57:12.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:57:12.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:57:12.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:57:12.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:57:12.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:57:12.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:57:12.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:57:12.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:57:12.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:57:12.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:57:12.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:57:12.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:57:12.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:57:12.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:57:12.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:57:12.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:57:12.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:57:12.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:57:12.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:57:12.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:57:12.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:57:12.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:57:12.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:57:12.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:57:12.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:57:12.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:57:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:57:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:57:12.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:57:12.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:57:12.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:57:12.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:57:12.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:57:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:57:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:57:12.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:57:12.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:57:12.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:57:12.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:57:12.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:57:12.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6369: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6370: TEST_request_scrub_priority: local poolname=psr_pool 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6371: TEST_request_scrub_priority: local objname=POBJ 2026-02-20T16:57:12.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6372: TEST_request_scrub_priority: local OBJECTS=64 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6373: TEST_request_scrub_priority: local PGS=8 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6375: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:57:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:57:12.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:57:12.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:57:12.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:57:12.864 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:57:12.864 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.864 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:57:12.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:57:12.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:57:12.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:57:12.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:57:12.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:57:12.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:57:12.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:57:12.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:57:12.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:57:12.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:57:12.898 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:57:12.899 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.899 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.899 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:57:12.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:57:12.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:57:12.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:57:12.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:57:12.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:57:12.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:57:12.957 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:12.958 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:12.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:57:12.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:57:12.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6376: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:57:13.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:57:13.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:57:13.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:57:13.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:57:13.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:57:13.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:13.112 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:13.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:57:13.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:57:13.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:57:13.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6377: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-20T16:57:13.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6378: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2026-02-20T16:57:13.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6379: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-20T16:57:13.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:57:13.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:57:13.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:57:13.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=306d305c-0673-4d5c-aba4-af8d99968ddd 2026-02-20T16:57:13.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 306d305c-0673-4d5c-aba4-af8d99968ddd' 2026-02-20T16:57:13.137 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 306d305c-0673-4d5c-aba4-af8d99968ddd 2026-02-20T16:57:13.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:57:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpkphp9R/ACBAAW+Kk0xBuOPao9ZinVCK72A== 2026-02-20T16:57:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpkphp9R/ACBAAW+Kk0xBuOPao9ZinVCK72A=="}' 2026-02-20T16:57:13.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 306d305c-0673-4d5c-aba4-af8d99968ddd -i td/osd-scrub-repair/0/new.json 2026-02-20T16:57:13.235 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:57:13.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:57:13.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBpkphp9R/ACBAAW+Kk0xBuOPao9ZinVCK72A== --osd-uuid 306d305c-0673-4d5c-aba4-af8d99968ddd 2026-02-20T16:57:13.261 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:13.259+0000 7f83c2475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:13.263 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:13.259+0000 7f83c2475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:13.264 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:13.263+0000 7f83c2475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:13.264 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:13.263+0000 7f83c2475a40 -1 bdev(0x55b2f2fa0c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:57:13.264 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:13.263+0000 7f83c2475a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:57:14.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:57:14.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:57:14.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:57:14.057 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:57:14.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:57:14.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:57:14.189 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:14.187+0000 7fdb77b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:14.192 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:14.191+0000 7fdb77b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:14.193 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:14.191+0000 7fdb77b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:57:14.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:57:14.259 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:57:14.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:57:14.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:57:14.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:57:14.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:57:14.617 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:14.615+0000 7fdb77b10a40 -1 Falling back to public interface 2026-02-20T16:57:15.321 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:57:15.319+0000 7fdb77b10a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:57:15.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:57:15.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:57:15.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:57:15.354 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:57:15.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:57:15.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:57:15.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:57:16.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:57:16.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:57:16.446 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:57:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:57:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:57:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:57:16.605 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1587536131,v1:127.0.0.1:6801/1587536131] [v2:127.0.0.1:6802/1587536131,v1:127.0.0.1:6803/1587536131] exists,up 306d305c-0673-4d5c-aba4-af8d99968ddd 2026-02-20T16:57:16.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:57:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:57:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:57:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6381: TEST_request_scrub_priority: create_pool psr_pool 8 8 2026-02-20T16:57:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2026-02-20T16:57:16.814 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool' created 2026-02-20T16:57:16.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:57:17.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6382: TEST_request_scrub_priority: wait_for_clean 2026-02-20T16:57:17.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:57:17.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:57:17.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:57:17.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:57:17.830 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:57:17.831 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:57:17.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:57:17.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:57:17.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:57:17.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:57:18.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:57:18.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:57:18.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:18.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:57:18.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-20T16:57:18.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-20T16:57:18.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-20T16:57:18.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:18.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-20T16:57:18.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:18.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:57:18.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-20T16:57:18.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:18.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-20T16:57:18.191 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836482 2026-02-20T16:57:18.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-20T16:57:18.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:18.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-20T16:57:18.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:19.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:57:19.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:19.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-20T16:57:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:57:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:19.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:19.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-20T16:57:19.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:57:19.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:57:19.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:57:19.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:57:19.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:57:19.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:57:19.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:57:19.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-20T16:57:19.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:57:19.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:57:19.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6384: TEST_request_scrub_priority: local osd=0 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6385: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2026-02-20T16:57:20.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:57:20.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2026-02-20T16:57:20.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2026-02-20T16:57:20.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:57:20.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:57:20.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:57:20.307 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:57:20.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:57:20.508 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:57:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:57:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:57:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2026-02-20T16:57:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: get_primary psr_pool POBJ 2026-02-20T16:57:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2026-02-20T16:57:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2026-02-20T16:57:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2026-02-20T16:57:20.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:57:20.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: local primary=0 2026-02-20T16:57:20.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: get_pg psr_pool POBJ 2026-02-20T16:57:20.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2026-02-20T16:57:20.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2026-02-20T16:57:20.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2026-02-20T16:57:20.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:57:20.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: local pg=1.1 2026-02-20T16:57:20.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: ceph osd dump 2026-02-20T16:57:20.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2026-02-20T16:57:20.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: awk '{ print $2 }' 2026-02-20T16:57:21.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: poolid=1 2026-02-20T16:57:21.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6390: TEST_request_scrub_priority: local otherpgs 2026-02-20T16:57:21.017 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: expr 8 - 1 2026-02-20T16:57:21.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: seq 0 7 2026-02-20T16:57:21.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.0 2026-02-20T16:57:21.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2026-02-20T16:57:21.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 ' 2026-02-20T16:57:21.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:21.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:21.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:21.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:21.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:21.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:21.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2026-02-20T16:57:21.244 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:21.244 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:21.245 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:21.245 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:21.244953+0000" 2026-02-20T16:57:21.245 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.1 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6395: TEST_request_scrub_priority: continue 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.2 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2026-02-20T16:57:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2026-02-20T16:57:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:21.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:21.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:21.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2026-02-20T16:57:21.481 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:21.481 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:21.481 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:21.481 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:21.481780+0000" 2026-02-20T16:57:21.482 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:21.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.3 2026-02-20T16:57:21.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2026-02-20T16:57:21.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2026-02-20T16:57:21.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:21.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:21.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:21.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:21.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:21.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:21.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2026-02-20T16:57:21.716 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:21.716 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:21.717 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:21.717 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:21.717068+0000" 2026-02-20T16:57:21.717 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:21.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.4 2026-02-20T16:57:21.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2026-02-20T16:57:21.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2026-02-20T16:57:21.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:21.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:21.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:21.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:21.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:21.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:21.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2026-02-20T16:57:21.949 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:21.949 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:21.950 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:21.950 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:21.949854+0000" 2026-02-20T16:57:21.950 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:21.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.5 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:21.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:21.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:21.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:22.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:22.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2026-02-20T16:57:22.185 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:22.185 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:22.185 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:22.185 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:22.185641+0000" 2026-02-20T16:57:22.185 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:22.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:22.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.6 2026-02-20T16:57:22.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2026-02-20T16:57:22.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2026-02-20T16:57:22.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:22.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:22.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:22.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:22.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:22.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:22.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2026-02-20T16:57:22.417 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:22.417 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:22.417 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:22.418 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:22.417854+0000" 2026-02-20T16:57:22.418 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:22.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-20T16:57:22.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.7 2026-02-20T16:57:22.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2026-02-20T16:57:22.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2026-02-20T16:57:22.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:22.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:22.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:22.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:22.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:22.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:22.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2026-02-20T16:57:22.656 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:22.656 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:22.657 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:57:22.657 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:57:22.656922+0000" 2026-02-20T16:57:22.657 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:22.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6403: TEST_request_scrub_priority: sleep 15 2026-02-20T16:57:37.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6404: TEST_request_scrub_priority: flush_pg_stats 2026-02-20T16:57:37.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:57:37.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:57:37.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:57:37.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:57:37.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:57:37.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:57:37.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-20T16:57:37.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-20T16:57:37.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-20T16:57:37.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:57:37.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-20T16:57:37.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:57:37.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:57:37.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-20T16:57:37.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:57:37.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-20T16:57:37.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-20T16:57:37.905 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836487 2026-02-20T16:57:37.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:38.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:57:38.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:39.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:57:39.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:39.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:57:39.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:57:40.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:57:40.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:57:40.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-20T16:57:40.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-20T16:57:40.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:40.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:40.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:40.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:40.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:40.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6408: TEST_request_scrub_priority: ceph tell 1.1 scrub 2026-02-20T16:57:40.613 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:57:40.613 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:57:40.613 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T16:57:40.614 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T16:57:40.614 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:57:40.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6410: TEST_request_scrub_priority: ceph osd unset noscrub 2026-02-20T16:57:40.824 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is unset 2026-02-20T16:57:40.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6411: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2026-02-20T16:57:41.030 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is unset 2026-02-20T16:57:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6413: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:41.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:41.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:41.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:41.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:41.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:57:16.813428+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:41.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:42.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:42.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:42.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-20T16:57:42.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:42.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:42.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:42.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:42.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:57:16.813428+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:42.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:43.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:43.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:43.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-20T16:57:43.362 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:57:43.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:43.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:57:43.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:43.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:57:41.327953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:43.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:57:43.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.0 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:43.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:43.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:43.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:44.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:44.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:44.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:44.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:44.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:44.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:44.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:44.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:44.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:45.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:45.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:45.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:45.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:45.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:45.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:45.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:45.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:45.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:46.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:46.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:46.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:46.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:46.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:46.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:46.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:47.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:47.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:48.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:48.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:48.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:48.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:48.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:48.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:48.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:48.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:48.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:49.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:49.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:49.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:49.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:49.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:49.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:49.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:49.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:49.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:50.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:50.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:50.472 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:50.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:50.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:51.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:51.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:51.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:51.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:51.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:51.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:51.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:51.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:51.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:52.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:52.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:52.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:52.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:52.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:52.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:52.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:52.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:52.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:53.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:53.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:53.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:53.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:53.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:53.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:53.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:54.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:54.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:55.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:55.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:55.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:55.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:55.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:55.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:55.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:55.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:55.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:56.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:56.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:56.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:56.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:56.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:56.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:56.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:56.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:56.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:57.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:57.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:57.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:57.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:57.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:57.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:57.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:57.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:57.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:58.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:58.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:58.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:58.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:58.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:58.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:58.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:58.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:58.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:57:59.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:57:59.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:57:59.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:57:59.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:57:59.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:57:59.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:57:59.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:57:59.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:57:59.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:00.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:00.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:00.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:01.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:01.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:02.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:02.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:02.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:02.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:02.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:02.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:02.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:02.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:02.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:03.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:03.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:03.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:03.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:03.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:03.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:03.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:03.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:03.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:04.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:04.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:04.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:04.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:04.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:04.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:04.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:04.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:04.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:05.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:05.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:05.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:05.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:05.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:05.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:05.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:06.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:06.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:06.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:06.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:06.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:06.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:07.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:07.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:07.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:07.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:07.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:07.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:07.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:08.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:08.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:09.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:09.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:09.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:09.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:09.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:09.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:09.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:09.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:09.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:10.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:10.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:10.209 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:10.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:10.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:10.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:10.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:10.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:10.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:11.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:11.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:11.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:11.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:11.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:11.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:11.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:11.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:11.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:12.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:12.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:12.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:12.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:12.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:12.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:12.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:12.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:12.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:13.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:13.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:13.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:13.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:13.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:13.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:13.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:13.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:13.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:14.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:14.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:14.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:14.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:14.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:14.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:15.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:15.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:16.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:16.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:16.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:16.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:16.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:16.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:16.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:17.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:17.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:17.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:17.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:17.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:17.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:17.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:17.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:17.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:18.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:18.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:18.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:18.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:18.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:18.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:18.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:18.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:18.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:19.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:19.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:19.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:19.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:19.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:19.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:19.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:19.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:19.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:20.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:20.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:20.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:20.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:20.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:20.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:20.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:20.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:20.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:21.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:21.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:21.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:21.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:21.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:21.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:21.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:21.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:21.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:22.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:22.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:22.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:22.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:22.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:22.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:22.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:23.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.244953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:23.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:24.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:24.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:24.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T16:58:24.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T16:58:24.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:24.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:24.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T16:58:24.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:21.305771+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:24.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.2 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.2 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-20T16:58:24.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.481780+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:24.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:25.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:25.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:25.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-20T16:58:25.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-20T16:58:25.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:25.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-20T16:58:25.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:25.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.481780+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:25.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:26.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:26.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:26.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-20T16:58:26.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-20T16:58:26.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:26.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:26.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-20T16:58:26.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:21.481780+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:26.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:27.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:27.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:27.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-20T16:58:27.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-20T16:58:27.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:27.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:27.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-20T16:58:27.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:22.327555+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.3 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:27.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-20T16:58:27.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:58:27.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:27.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:27.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:58:28.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:24.313804+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.4 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.4 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2026-02-20T16:58:28.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:28.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2026-02-20T16:58:28.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:25.289519+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.5 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.5 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-20T16:58:28.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:28.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:28.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-20T16:58:28.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:26.255567+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.6 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.6 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:28.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:28.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:28.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:28.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:28.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:28.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:28.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:22.417854+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:28.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:29.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:29.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:29.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:29.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:29.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:29.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:29.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:29.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:22.417854+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:29.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:30.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:30.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:30.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:30.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:30.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:30.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:30.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:30.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:22.417854+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:30.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:31.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:31.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:31.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:31.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:31.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:31.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:31.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:32.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:22.417854+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:32.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:33.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:33.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:33.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:33.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:33.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:33.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:33.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:33.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T16:57:22.417854+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:33.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T16:58:34.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T16:58:34.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:34.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-20T16:58:34.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-20T16:58:34.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:34.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:34.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-20T16:58:34.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:27.264353+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.7 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.7 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2026-02-20T16:58:34.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:34.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:34.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:58:28.243624+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T16:58:34.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:58:34.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-20T16:58:34.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T16:57:41.327953+0000 '>' 2026-02-20T16:57:16.813428+0000 2026-02-20T16:58:34.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T16:58:34.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2026-02-20T16:58:34.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -v purged_snaps 2026-02-20T16:58:34.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: head -1 2026-02-20T16:58:34.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2026-02-20T16:58:34.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -q 1.1 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:58:34.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:58:34.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:58:34.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:58:34.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:58:34.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:58:34.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:58:34.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:58:34.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:58:34.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:58:34.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:58:34.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:58:34.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:58:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:58:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:58:34.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:58:34.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:58:34.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:58:34.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:58:34.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:58:34.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:58:34.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:58:34.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:58:34.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:58:34.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:58:34.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:58:34.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:58:34.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:58:34.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:58:34.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:58:34.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:58:34.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:58:34.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:58:34.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:58:34.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:58:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:58:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:58:34.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:58:34.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:58:34.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:58:34.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:58:34.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:58:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:58:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:58:34.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6011: TEST_scrub_warning: local dir=td/osd-scrub-repair 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_scrub_warning: local poolname=psr_pool 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6013: TEST_scrub_warning: local objname=POBJ 2026-02-20T16:58:34.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6014: TEST_scrub_warning: local scrubs=5 2026-02-20T16:58:34.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6015: TEST_scrub_warning: local deep_scrubs=5 2026-02-20T16:58:34.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6016: TEST_scrub_warning: local i1_day=86400 2026-02-20T16:58:34.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: calc 86400 '*' 7 2026-02-20T16:58:34.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 7}' 2026-02-20T16:58:34.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: local i7_days=604800 2026-02-20T16:58:34.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: calc 86400 '*' 14 2026-02-20T16:58:34.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 14}' 2026-02-20T16:58:34.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: local i14_days=1209600 2026-02-20T16:58:34.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6019: TEST_scrub_warning: local overdue=0.5 2026-02-20T16:58:34.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2026-02-20T16:58:34.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2026-02-20T16:58:34.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: local conf_overdue_seconds=993600 2026-02-20T16:58:34.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2026-02-20T16:58:34.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: local pool_overdue_seconds=1900800 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6023: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:58:34.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:58:34.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:58:34.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:58:34.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:58:34.882 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:58:34.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:58:34.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-20T16:58:34.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:58:34.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:58:34.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:58:34.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:58:34.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:58:34.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:58:34.915 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:58:34.915 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:58:34.915 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:58:34.915 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:58:34.916 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.916 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.916 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:58:34.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:58:34.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:58:34.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:58:34.979 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:58:34.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:58:34.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:34.980 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:34.980 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:58:34.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:58:34.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:58:35.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6024: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:58:35.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:58:35.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:58:35.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:58:35.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:58:35.127 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:58:35.127 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:35.127 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:35.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:58:35.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:58:35.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-20T16:58:35.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6025: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:58:35.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:58:35.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2026-02-20T16:58:35.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:58:35.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:58:35.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1b2fb4a0-918e-47a9-86d2-ea46e3aa2723 2026-02-20T16:58:35.152 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 1b2fb4a0-918e-47a9-86d2-ea46e3aa2723 2026-02-20T16:58:35.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1b2fb4a0-918e-47a9-86d2-ea46e3aa2723' 2026-02-20T16:58:35.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:58:35.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7kphpiZ7MCRAA7vir8Hqb1AG5p9Q3W1Vx8Q== 2026-02-20T16:58:35.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7kphpiZ7MCRAA7vir8Hqb1AG5p9Q3W1Vx8Q=="}' 2026-02-20T16:58:35.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1b2fb4a0-918e-47a9-86d2-ea46e3aa2723 -i td/osd-scrub-repair/0/new.json 2026-02-20T16:58:35.258 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:58:35.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:58:35.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQC7kphpiZ7MCRAA7vir8Hqb1AG5p9Q3W1Vx8Q== --osd-uuid 1b2fb4a0-918e-47a9-86d2-ea46e3aa2723 2026-02-20T16:58:35.283 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:35.277+0000 7f3cb3ae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:35.284 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:35.281+0000 7f3cb3ae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:35.286 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:35.281+0000 7f3cb3ae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:35.286 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:35.281+0000 7f3cb3ae8a40 -1 bdev(0x55754873cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:58:35.286 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:35.281+0000 7f3cb3ae8a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:58:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:58:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:58:36.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:58:36.025 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:58:36.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:58:36.127 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:58:36.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:58:36.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2026-02-20T16:58:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:58:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:58:36.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:58:36.154 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:36.149+0000 7f02e8675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:36.156 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:36.153+0000 7f02e8675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:36.157 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:36.153+0000 7f02e8675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:58:36.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:58:36.222 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:58:36.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:58:36.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:58:36.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:58:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:58:36.606 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:36.601+0000 7f02e8675a40 -1 Falling back to public interface 2026-02-20T16:58:37.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:58:37.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:58:37.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:58:37.312 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:58:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:58:37.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:58:37.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:58:37.601 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:58:37.597+0000 7f02e8675a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:58:38.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:58:38.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:58:38.424 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:58:38.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:58:38.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:58:38.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:58:38.583 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2429276329,v1:127.0.0.1:6801/2429276329] [v2:127.0.0.1:6802/2429276329,v1:127.0.0.1:6803/2429276329] exists,up 1b2fb4a0-918e-47a9-86d2-ea46e3aa2723 2026-02-20T16:58:38.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:58:38.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:58:38.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:58:38.585 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: expr 5 + 5 2026-02-20T16:58:38.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: seq 1 10 2026-02-20T16:58:38.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:38.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-1 1 1 2026-02-20T16:58:38.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2026-02-20T16:58:38.765 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-1' created 2026-02-20T16:58:38.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:39.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:39.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:39.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:39.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:39.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:39.779 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:39.779 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:39.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:39.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:39.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:39.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:40.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:40.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:40.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-20T16:58:40.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-20T16:58:40.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-20T16:58:40.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:40.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-20T16:58:40.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:40.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:40.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-20T16:58:40.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:40.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-20T16:58:40.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-20T16:58:40.144 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836482 2026-02-20T16:58:40.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:40.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-20T16:58:40.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:41.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:58:41.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:41.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-20T16:58:41.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:58:41.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:41.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:41.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T16:58:41.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:58:41.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:58:41.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:58:41.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:58:41.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:58:41.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:58:41.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:58:41.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T16:58:41.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:58:41.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:41.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:41.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T16:58:41.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:58:41.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:58:41.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-20T16:58:41.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6033: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2026-02-20T16:58:42.195 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 scrub_max_interval to 1209600 2026-02-20T16:58:42.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:58:42.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 1 = 6 ']' 2026-02-20T16:58:42.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:42.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-2 1 1 2026-02-20T16:58:42.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2026-02-20T16:58:42.400 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-2' created 2026-02-20T16:58:42.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:43.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:43.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:43.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:43.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:43.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:43.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:43.415 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:43.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:43.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:43.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:43.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:43.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:43.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:43.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:43.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:43.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:43.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:43.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:43.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:43.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:43.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T16:58:43.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T16:58:43.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T16:58:43.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:43.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T16:58:43.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:43.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:43.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T16:58:43.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:43.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T16:58:43.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T16:58:43.779 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T16:58:43.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:43.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:58:43.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:44.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:58:44.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:45.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T16:58:45.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:46.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:58:46.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T16:58:46.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:58:46.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:46.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:46.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 2 == 0 2026-02-20T16:58:46.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:58:46.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:58:46.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=2 2026-02-20T16:58:46.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:58:46.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:46.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 2 = 2 2026-02-20T16:58:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:58:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:58:46.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-20T16:58:46.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:58:46.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 2 = 6 ']' 2026-02-20T16:58:46.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:46.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-3 1 1 2026-02-20T16:58:46.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2026-02-20T16:58:47.012 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-3' created 2026-02-20T16:58:47.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:48.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:48.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:48.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:48.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:48.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:48.026 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:48.026 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:48.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:48.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:48.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:48.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:48.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:48.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:48.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:48.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:48.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:48.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:48.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:48.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:48.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:48.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-20T16:58:48.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-20T16:58:48.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-20T16:58:48.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:48.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-20T16:58:48.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:48.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:48.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-20T16:58:48.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:48.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-20T16:58:48.390 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836485 2026-02-20T16:58:48.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-20T16:58:48.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:48.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-20T16:58:48.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:49.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:58:49.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:49.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-20T16:58:49.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:58:49.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:49.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:49.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 3 == 0 2026-02-20T16:58:49.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:58:49.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:58:50.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-20T16:58:50.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:58:50.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:50.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:50.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 3 2026-02-20T16:58:50.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:58:50.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:58:50.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-20T16:58:50.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:58:50.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 3 = 6 ']' 2026-02-20T16:58:50.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:50.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-4 1 1 2026-02-20T16:58:50.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2026-02-20T16:58:50.477 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-4' created 2026-02-20T16:58:50.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:51.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:51.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:51.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:51.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:51.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:51.491 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:51.491 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:51.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:51.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:51.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:51.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:51.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:51.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:51.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:51.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:51.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:51.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-20T16:58:51.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-20T16:58:51.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-20T16:58:51.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:51.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-20T16:58:51.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:51.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-20T16:58:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:51.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-20T16:58:51.844 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836487 2026-02-20T16:58:51.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-20T16:58:51.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:52.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:58:52.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:53.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:58:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:53.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-20T16:58:53.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:54.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:58:54.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:54.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-20T16:58:54.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:58:54.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:54.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:54.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:58:54.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:58:54.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:58:54.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:58:54.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:54.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:54.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:58:54.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:58:54.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:58:54.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-20T16:58:54.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:58:54.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 4 = 6 ']' 2026-02-20T16:58:54.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:54.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-5 1 1 2026-02-20T16:58:54.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2026-02-20T16:58:55.112 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-5' created 2026-02-20T16:58:55.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:56.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:56.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:56.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:56.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:56.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:56.127 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:56.128 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:56.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:56.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:56.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:56.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:56.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:56.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:56.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:56.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:56.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:56.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:56.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:56.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:56.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:56.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-20T16:58:56.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-20T16:58:56.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-20T16:58:56.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:56.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-20T16:58:56.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:56.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:56.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-20T16:58:56.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:56.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-20T16:58:56.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-20T16:58:56.490 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836489 2026-02-20T16:58:56.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:56.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-20T16:58:56.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:58:57.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:58:57.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:58:57.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-20T16:58:57.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:58:57.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:57.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:58.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-20T16:58:58.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:58:58.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:58:58.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-20T16:58:58.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:58:58.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:58:58.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:58:58.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-20T16:58:58.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:58:58.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:58:58.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-20T16:58:58.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:58:58.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 5 = 6 ']' 2026-02-20T16:58:58.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:58:58.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-6 1 1 2026-02-20T16:58:58.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2026-02-20T16:58:58.574 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-6' created 2026-02-20T16:58:58.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:58:59.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:58:59.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:58:59.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:58:59.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:58:59.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:58:59.589 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:58:59.590 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:58:59.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:58:59.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:58:59.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:58:59.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:58:59.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:58:59.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:58:59.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:58:59.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:58:59.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:58:59.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:58:59.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:58:59.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:58:59.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:58:59.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T16:58:59.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T16:58:59.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T16:58:59.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:58:59.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T16:58:59.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:58:59.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:58:59.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T16:58:59.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:58:59.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T16:58:59.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T16:58:59.952 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T16:58:59.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:00.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:59:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:01.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:01.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:01.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-20T16:59:01.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:02.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:59:02.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:02.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-20T16:59:02.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:02.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:02.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:02.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:02.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:02.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:02.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-20T16:59:02.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:02.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:02.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:02.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-20T16:59:02.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:02.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:02.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 6 = 1 ']' 2026-02-20T16:59:02.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:02.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-20T16:59:02.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6037: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2026-02-20T16:59:03.189 INFO:tasks.workunit.client.0.trial013.stderr:set pool 6 deep_scrub_interval to 1209600 2026-02-20T16:59:03.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:03.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-7 1 1 2026-02-20T16:59:03.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2026-02-20T16:59:03.394 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-7' created 2026-02-20T16:59:03.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:59:04.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:59:04.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:04.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:04.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:04.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:04.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:04.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:04.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:04.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:04.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:04.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:04.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:59:04.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:04.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:04.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T16:59:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T16:59:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T16:59:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:04.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T16:59:04.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:04.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:04.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T16:59:04.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:04.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T16:59:04.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T16:59:04.763 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836493 2026-02-20T16:59:04.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:04.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T16:59:04.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:05.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:05.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:06.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T16:59:06.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:06.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:06.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:06.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 7 == 0 2026-02-20T16:59:06.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:06.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:06.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:06.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=7 2026-02-20T16:59:06.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:06.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:06.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:06.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 7 = 7 2026-02-20T16:59:06.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:06.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:06.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 7 = 1 ']' 2026-02-20T16:59:06.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:06.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-20T16:59:06.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:06.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-8 1 1 2026-02-20T16:59:06.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2026-02-20T16:59:06.836 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-8' created 2026-02-20T16:59:06.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:59:07.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:59:07.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:07.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:07.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:07.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:07.851 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:07.851 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:07.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:07.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:07.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:07.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:07.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:08.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:59:08.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:08.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:08.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:08.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836494 2026-02-20T16:59:08.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836494 2026-02-20T16:59:08.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836494' 2026-02-20T16:59:08.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:08.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836494 2026-02-20T16:59:08.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:08.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:08.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836494 2026-02-20T16:59:08.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:08.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836494 2026-02-20T16:59:08.207 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836494 2026-02-20T16:59:08.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-20T16:59:08.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:08.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836494 2026-02-20T16:59:08.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:09.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:09.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:09.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836494 2026-02-20T16:59:09.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:09.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:09.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:09.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-20T16:59:09.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:09.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:09.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-20T16:59:09.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:09.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:09.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:10.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-20T16:59:10.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:10.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:10.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 8 = 1 ']' 2026-02-20T16:59:10.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:10.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-20T16:59:10.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:10.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-9 1 1 2026-02-20T16:59:10.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2026-02-20T16:59:10.267 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-9' created 2026-02-20T16:59:10.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:59:11.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:59:11.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:11.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:11.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:11.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:11.283 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:11.283 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:11.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:11.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:11.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:11.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:11.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:11.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:59:11.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:11.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:11.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:11.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-20T16:59:11.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-20T16:59:11.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-20T16:59:11.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:11.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-20T16:59:11.645 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:11.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:11.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-20T16:59:11.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:11.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-20T16:59:11.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-20T16:59:11.649 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836496 2026-02-20T16:59:11.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:11.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-20T16:59:11.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:12.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:12.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-20T16:59:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:13.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:59:13.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:14.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-20T16:59:14.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:14.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:14.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:14.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 9 == 0 2026-02-20T16:59:14.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:14.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:14.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:14.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:14.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:14.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:14.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:14.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=9 2026-02-20T16:59:14.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:14.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:14.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:14.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 9 = 9 2026-02-20T16:59:14.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:14.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:14.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 9 = 1 ']' 2026-02-20T16:59:14.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:14.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-20T16:59:14.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:14.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-10 1 1 2026-02-20T16:59:14.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2026-02-20T16:59:14.894 INFO:tasks.workunit.client.0.trial013.stderr:pool 'psr_pool-10' created 2026-02-20T16:59:14.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:59:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-20T16:59:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:15.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:15.908 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:15.909 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:15.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:15.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:15.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:16.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:16.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:59:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:16.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:16.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-20T16:59:16.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-20T16:59:16.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498' 2026-02-20T16:59:16.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:16.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-20T16:59:16.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:16.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:16.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836498 2026-02-20T16:59:16.264 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836498 2026-02-20T16:59:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-20T16:59:16.266 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836498 2026-02-20T16:59:16.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:16.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836498 2026-02-20T16:59:16.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:17.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:17.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:17.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836498 2026-02-20T16:59:17.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:17.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:17.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 10 == 0 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:17.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:17.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:17.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:17.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=10 2026-02-20T16:59:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 10 = 10 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 10 = 1 ']' 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6042: TEST_scrub_warning: local primary=0 2026-02-20T16:59:18.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6044: TEST_scrub_warning: ceph osd set noscrub 2026-02-20T16:59:18.350 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:59:18.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6045: TEST_scrub_warning: ceph osd set nodeep-scrub 2026-02-20T16:59:18.554 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:59:18.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6046: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2026-02-20T16:59:18.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6047: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2026-02-20T16:59:18.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6048: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2026-02-20T16:59:19.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6049: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2026-02-20T16:59:19.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: seq 1 5 2026-02-20T16:59:19.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-20T16:59:19.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-20T16:59:19.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6056: TEST_scrub_warning: overdue_seconds=1900800 2026-02-20T16:59:19.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 1900800 + 100 2026-02-20T16:59:19.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2026-02-20T16:59:19.309 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.309 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:19.309 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.309 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-01-29T16:57:39.309109+0000" 2026-02-20T16:59:19.309 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-20T16:59:19.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-20T16:59:19.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 200 2026-02-20T16:59:19.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2026-02-20T16:59:19.385 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.385 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:19.385 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.385 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:55:59.384968+0000" 2026-02-20T16:59:19.385 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-20T16:59:19.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-20T16:59:19.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 300 2026-02-20T16:59:19.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2026-02-20T16:59:19.461 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.461 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:19.461 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.461 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:54:19.460881+0000" 2026-02-20T16:59:19.461 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-20T16:59:19.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-20T16:59:19.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 400 2026-02-20T16:59:19.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2026-02-20T16:59:19.536 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.537 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:19.537 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.537 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:52:39.536565+0000" 2026-02-20T16:59:19.537 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-20T16:59:19.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-20T16:59:19.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 500 2026-02-20T16:59:19.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2026-02-20T16:59:19.612 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.613 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:19.613 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.613 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:50:59.612643+0000" 2026-02-20T16:59:19.613 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.623 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.624 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 5 2026-02-20T16:59:19.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: seq 6 10 2026-02-20T16:59:19.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:19.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-20T16:59:19.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6067: TEST_scrub_warning: overdue_seconds=1900800 2026-02-20T16:59:19.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 1900800 + 600 2026-02-20T16:59:19.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2026-02-20T16:59:19.695 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.695 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:59:19.695 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.695 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-01-29T16:49:19.694746+0000" 2026-02-20T16:59:19.695 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:19.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-20T16:59:19.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 700 2026-02-20T16:59:19.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2026-02-20T16:59:19.769 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.769 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:59:19.769 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.770 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:47:39.769089+0000" 2026-02-20T16:59:19.770 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:19.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-20T16:59:19.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 800 2026-02-20T16:59:19.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2026-02-20T16:59:19.844 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.845 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:59:19.845 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.845 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:45:59.844836+0000" 2026-02-20T16:59:19.845 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:19.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-20T16:59:19.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 900 2026-02-20T16:59:19.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2026-02-20T16:59:19.920 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.920 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:59:19.920 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.920 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:44:19.919983+0000" 2026-02-20T16:59:19.920 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:19.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-20T16:59:19.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-20T16:59:19.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-20T16:59:19.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-20T16:59:19.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 1000 2026-02-20T16:59:19.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2026-02-20T16:59:19.994 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:19.994 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T16:59:19.994 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:19.995 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T04:42:39.994342+0000" 2026-02-20T16:59:19.995 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:20.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6073: TEST_scrub_warning: flush_pg_stats 2026-02-20T16:59:20.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:20.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:20.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T16:59:20.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:20.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:20.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:20.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-20T16:59:20.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-20T16:59:20.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-20T16:59:20.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:20.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-20T16:59:20.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:20.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:20.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-20T16:59:20.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:20.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-20T16:59:20.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-20T16:59:20.249 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836500 2026-02-20T16:59:20.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:20.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836500 2026-02-20T16:59:20.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:21.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:21.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:21.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-20T16:59:21.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6075: TEST_scrub_warning: ceph health 2026-02-20T16:59:21.748 INFO:tasks.workunit.client.0.trial013.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-20T16:59:21.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6076: TEST_scrub_warning: ceph health detail 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 10.0 not deep-scrubbed since 2026-02-09T04:42:39.994342+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 9.0 not deep-scrubbed since 2026-02-09T04:44:19.919983+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 8.0 not deep-scrubbed since 2026-02-09T04:45:59.844836+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 7.0 not deep-scrubbed since 2026-02-09T04:47:39.769089+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 6.0 not deep-scrubbed since 2026-01-29T16:49:19.694746+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout:[WRN] PG_NOT_SCRUBBED: 5 pgs not scrubbed in time 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 5.0 not scrubbed since 2026-02-09T04:50:59.612643+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 4.0 not scrubbed since 2026-02-09T04:52:39.536565+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 3.0 not scrubbed since 2026-02-09T04:54:19.460881+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 2.0 not scrubbed since 2026-02-09T04:55:59.384968+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pg 1.0 not scrubbed since 2026-01-29T16:57:39.309109+0000 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-1' has no replicas configured 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-2' has no replicas configured 2026-02-20T16:59:21.944 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-3' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-4' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-5' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-6' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-7' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-8' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-9' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout: pool 'psr_pool-10' has no replicas configured 2026-02-20T16:59:21.945 INFO:tasks.workunit.client.0.trial013.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2026-02-20T16:59:21.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: ceph health 2026-02-20T16:59:21.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2026-02-20T16:59:22.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: ceph health 2026-02-20T16:59:22.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2026-02-20T16:59:22.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: ceph health detail 2026-02-20T16:59:22.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: grep 'not scrubbed since' 2026-02-20T16:59:22.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: wc -l 2026-02-20T16:59:22.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: COUNT=5 2026-02-20T16:59:22.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: expr 5+5 2026-02-20T16:59:22.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: (( 5 != 5 && 5 != 5+5 )) 2026-02-20T16:59:22.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: ceph health detail 2026-02-20T16:59:22.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: grep 'not deep-scrubbed since' 2026-02-20T16:59:22.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: wc -l 2026-02-20T16:59:22.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: COUNT=5 2026-02-20T16:59:22.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6090: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2026-02-20T16:59:22.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:59:22.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:59:22.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:59:22.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:59:22.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:59:22.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:59:22.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:59:22.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:59:22.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:59:22.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:59:22.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:59:22.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:59:22.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:59:22.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:59:22.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:59:22.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:59:22.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:22.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:59:22.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:59:22.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:22.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:59:22.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:59:22.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:59:22.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:59:22.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:22.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:22.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:59:22.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:59:22.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:59:22.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:59:22.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:59:22.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:59:22.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:59:22.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:59:22.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:59:22.857 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:59:22.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:22.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:59:22.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:59:22.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:22.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:59:22.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:59:22.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:59:22.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:59:22.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:22.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:22.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:59:22.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:59:22.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:59:22.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:59:22.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:59:22.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:22.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:22.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:59:22.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:160: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-20T16:59:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: local -A cluster_conf 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: TEST_skip_non_repair_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-20T16:59:22.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T16:59:22.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T16:59:22.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T16:59:23.161 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 1aa7b71a-adf4-4260-9cdf-e141389cd23f 2026-02-20T16:59:23.265 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:23.295 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:23.288+0000 7ff27dd45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:23.296 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:23.292+0000 7ff27dd45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:23.297 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:23.292+0000 7ff27dd45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:23.298 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:23.292+0000 7ff27dd45a40 -1 bdev(0x55c0f38ccc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:59:23.298 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:23.292+0000 7ff27dd45a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:59:24.032 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:59:24.134 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:59:24.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:24.160+0000 7f65541b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:24.169 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:24.164+0000 7f65541b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:24.171 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:24.164+0000 7f65541b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:24.227 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:24.685 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:24.680+0000 7f65541b6a40 -1 Falling back to public interface 2026-02-20T16:59:25.318 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:25.689 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:25.684+0000 7f65541b6a40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:59:26.446 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:59:26.603 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3331266255,v1:127.0.0.1:6801/3331266255] [v2:127.0.0.1:6802/3331266255,v1:127.0.0.1:6803/3331266255] exists,up 1aa7b71a-adf4-4260-9cdf-e141389cd23f 2026-02-20T16:59:26.610 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 133ccbaf-c267-4081-b416-ced021b135a7 2026-02-20T16:59:26.790 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:26.829 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:26.824+0000 7f2b7d7d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:26.830 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:26.824+0000 7f2b7d7d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:26.831 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:26.824+0000 7f2b7d7d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:26.832 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:26.824+0000 7f2b7d7d3a40 -1 bdev(0x560ae9416c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:59:26.832 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:26.824+0000 7f2b7d7d3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:59:27.520 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:59:27.728 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:59:27.760 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:27.756+0000 7f6a65c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:27.761 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:27.756+0000 7f6a65c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:27.762 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:27.756+0000 7f6a65c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:27.882 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:28.165 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:28.160+0000 7f6a65c41a40 -1 Falling back to public interface 2026-02-20T16:59:28.790 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:28.784+0000 7f6a65c41a40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:59:29.043 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:30.210 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:59:30.370 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3351896703,v1:127.0.0.1:6811/3351896703] [v2:127.0.0.1:6812/3351896703,v1:127.0.0.1:6813/3351896703] exists,up 133ccbaf-c267-4081-b416-ced021b135a7 2026-02-20T16:59:31.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: TEST_skip_non_repair_during_recovery: local poolname=rbd 2026-02-20T16:59:31.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:171: TEST_skip_non_repair_during_recovery: create_rbd_pool 2026-02-20T16:59:31.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-20T16:59:31.521 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' does not exist 2026-02-20T16:59:31.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-20T16:59:31.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-20T16:59:31.740 INFO:tasks.workunit.client.0.trial013.stderr:pool 'rbd' created 2026-02-20T16:59:31.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T16:59:32.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-20T16:59:33.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: TEST_skip_non_repair_during_recovery: wait_for_clean 2026-02-20T16:59:33.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:33.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:33.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:33.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:33.036 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:33.036 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:33.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:33.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:33.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:33.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:33.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:59:33.318 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:59:33.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:33.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:33.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:33.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T16:59:33.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T16:59:33.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T16:59:33.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:33.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:59:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T16:59:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T16:59:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-20T16:59:33.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:33.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T16:59:33.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:33.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:33.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T16:59:33.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:33.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T16:59:33.468 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T16:59:33.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T16:59:33.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:33.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T16:59:33.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:33.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T16:59:33.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:33.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:59:33.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T16:59:33.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:33.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T16:59:33.634 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T16:59:33.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T16:59:33.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:59:33.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705669 2026-02-20T16:59:33.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:34.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:34.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:59:34.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705669 2026-02-20T16:59:34.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:35.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T16:59:35.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:59:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705671 -lt 38654705669 2026-02-20T16:59:36.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:36.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:36.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:36.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:36.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:36.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:36.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:59:36.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:36.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:36.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:174: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T16:59:36.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T16:59:36.870 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T16:59:36.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T16:59:37.075 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T16:59:37.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T16:59:37.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T16:59:37.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T16:59:37.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-20T16:59:37.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:59:37.117 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T16:59:37.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T16:59:37.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-20T16:59:37.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T16:59:37.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2026-02-20T16:59:37.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2026-02-20T16:59:37.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: scrub_and_not_schedule: local poolname=rbd 2026-02-20T16:59:37.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: scrub_and_not_schedule: local osd=0 2026-02-20T16:59:37.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: get_pg rbd SOMETHING 2026-02-20T16:59:37.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-20T16:59:37.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T16:59:37.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-20T16:59:37.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T16:59:37.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: local pg=1.3 2026-02-20T16:59:37.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:59:37.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:59:37.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:59:37.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:59:37.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:59:37.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: local last_scrub=2026-02-20T16:59:31.739716+0000 2026-02-20T16:59:37.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:223: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2026-02-20T16:59:37.813 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T16:59:37.814 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T16:59:37.814 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T16:59:37.814 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T16:59:37.813674+0000" 2026-02-20T16:59:37.814 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T16:59:37.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: scrub_and_not_schedule: sleep 3 2026-02-20T16:59:40.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i=0 )) 2026-02-20T16:59:40.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-20T16:59:40.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:59:40.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:59:40.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:59:40.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:59:40.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:59:40.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-18T16:59:37.813674+0000 '>' 2026-02-20T16:59:31.739716+0000 2026-02-20T16:59:40.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-20T16:59:41.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-20T16:59:41.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-20T16:59:41.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:59:41.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:59:41.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:59:41.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:59:41.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:59:42.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-18T16:59:37.813674+0000 '>' 2026-02-20T16:59:31.739716+0000 2026-02-20T16:59:42.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-20T16:59:43.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-20T16:59:43.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-20T16:59:43.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-20T16:59:43.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-20T16:59:43.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T16:59:43.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T16:59:43.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-20T16:59:43.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-18T16:59:37.813674+0000 '>' 2026-02-20T16:59:31.739716+0000 2026-02-20T16:59:43.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-20T16:59:44.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-20T16:59:44.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-20T16:59:44.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:59:44.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T16:59:44.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T16:59:44.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T16:59:44.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T16:59:44.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T16:59:44.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:59:44.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:59:44.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:59:44.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:59:44.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:59:44.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-20T16:59:44.690 INFO:tasks.workunit.client.0.trial013.stdout:_ 2026-02-20T16:59:44.690 INFO:tasks.workunit.client.0.trial013.stdout:snapset 2026-02-20T16:59:44.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:59:44.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:44.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T16:59:44.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:59:44.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:start osd.0 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T16:59:44.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T16:59:44.813 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:44.811+0000 7f057aa49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:44.815 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:44.811+0000 7f057aa49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:44.815 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:44.815+0000 7f057aa49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:44.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:59:45.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:45.241 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:45.239+0000 7f057aa49a40 -1 Falling back to public interface 2026-02-20T16:59:45.907 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:45.903+0000 7f057aa49a40 -1 osd.0 20 log_to_monitors true 2026-02-20T16:59:46.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:46.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:46.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:59:46.115 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T16:59:46.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:46.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:59:46.275 INFO:tasks.workunit.client.0.trial013.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,20) [v2:127.0.0.1:6800/4129346639,v1:127.0.0.1:6801/4129346639] [v2:127.0.0.1:6802/4129346639,v1:127.0.0.1:6803/4129346639] exists,up 1aa7b71a-adf4-4260-9cdf-e141389cd23f 2026-02-20T16:59:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:59:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:59:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:59:46.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T16:59:46.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T16:59:46.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T16:59:46.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T16:59:46.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T16:59:46.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T16:59:46.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T16:59:46.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T16:59:46.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T16:59:46.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T16:59:46.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T16:59:46.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T16:59:46.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T16:59:46.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T16:59:46.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T16:59:46.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T16:59:46.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T16:59:46.560 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T16:59:46.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T16:59:46.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:46.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T16:59:46.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-20T16:59:46.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-20T16:59:46.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-20T16:59:46.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T16:59:46.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T16:59:46.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705683 2026-02-20T16:59:46.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705683 2026-02-20T16:59:46.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705683' 2026-02-20T16:59:46.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:46.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-20T16:59:46.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:46.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T16:59:46.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-20T16:59:46.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:46.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-20T16:59:46.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-20T16:59:46.710 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 98784247810 2026-02-20T16:59:46.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:46.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 98784247810 2026-02-20T16:59:46.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T16:59:47.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T16:59:47.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T16:59:48.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247810 2026-02-20T16:59:48.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T16:59:48.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705683 2026-02-20T16:59:48.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T16:59:48.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T16:59:48.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705683 2026-02-20T16:59:48.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T16:59:48.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705683 2026-02-20T16:59:48.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2026-02-20T16:59:48.043 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705683 2026-02-20T16:59:48.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T16:59:48.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705684 -lt 38654705683 2026-02-20T16:59:48.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T16:59:48.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:48.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:48.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-20T16:59:48.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T16:59:48.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T16:59:48.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T16:59:48.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T16:59:48.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T16:59:48.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T16:59:48.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T16:59:48.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-20T16:59:48.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T16:59:48.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T16:59:48.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T16:59:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-20T16:59:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T16:59:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T16:59:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:240: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-20T16:59:48.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-20T16:59:48.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T16:59:48.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:59:48.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:59:48.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:59:48.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:59:48.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:59:48.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:59:48.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:59:48.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:59:49.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:59:49.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:59:49.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:59:49.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:59:49.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:59:49.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:59:49.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:59:49.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:49.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:59:49.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:59:49.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:49.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:59:49.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:59:49.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:59:49.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:59:49.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T16:59:49.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T16:59:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T16:59:49.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T16:59:49.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T16:59:49.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T16:59:49.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T16:59:49.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T16:59:49.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T16:59:49.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T16:59:49.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T16:59:49.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T16:59:49.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T16:59:49.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T16:59:49.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T16:59:49.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:49.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T16:59:49.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T16:59:49.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T16:59:49.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T16:59:49.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T16:59:49.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T16:59:49.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T16:59:49.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T16:59:49.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T16:59:49.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T16:59:49.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T16:59:49.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T16:59:49.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T16:59:49.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T16:59:49.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T16:59:49.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T16:59:49.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T16:59:49.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:962: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=false 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T16:59:49.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T16:59:49.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T16:59:49.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:49.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:49.164 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:49.164 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.164 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:49.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T16:59:49.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T16:59:49.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T16:59:49.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:59:49.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:59:49.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T16:59:49.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T16:59:49.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:59:49.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:59:49.194 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:59:49.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T16:59:49.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T16:59:49.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T16:59:49.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T16:59:49.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T16:59:49.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T16:59:49.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T16:59:49.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T16:59:49.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T16:59:49.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T16:59:49.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T16:59:49.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T16:59:49.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T16:59:49.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:49.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:59:49.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T16:59:49.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:59:49.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:49.432 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:49.433 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:49.433 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:49.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:59:49.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3fab5dce-12db-482a-b3b2-263f112efe1d 2026-02-20T16:59:49.436 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 3fab5dce-12db-482a-b3b2-263f112efe1d 2026-02-20T16:59:49.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3fab5dce-12db-482a-b3b2-263f112efe1d' 2026-02-20T16:59:49.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:59:49.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFk5hpvIh4GhAAsWxuOiP0DYOhVkRl8KCDXg== 2026-02-20T16:59:49.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFk5hpvIh4GhAAsWxuOiP0DYOhVkRl8KCDXg=="}' 2026-02-20T16:59:49.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3fab5dce-12db-482a-b3b2-263f112efe1d -i td/osd-scrub-repair/0/new.json 2026-02-20T16:59:49.535 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:49.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T16:59:49.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFk5hpvIh4GhAAsWxuOiP0DYOhVkRl8KCDXg== --osd-uuid 3fab5dce-12db-482a-b3b2-263f112efe1d 2026-02-20T16:59:49.563 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:49.559+0000 7fd5e8b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:49.565 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:49.563+0000 7fd5e8b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:49.566 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:49.563+0000 7fd5e8b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:49.566 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:49.563+0000 7fd5e8b3fa40 -1 bdev(0x5571bfa82c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T16:59:49.566 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:49.563+0000 7fd5e8b3fa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T16:59:50.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T16:59:50.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:59:50.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T16:59:50.373 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T16:59:50.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:59:50.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T16:59:50.480 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T16:59:50.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:59:50.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:59:50.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:59:50.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:59:50.506 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:50.503+0000 7fa95c7cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:50.508 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:50.507+0000 7fa95c7cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:50.509 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:50.507+0000 7fa95c7cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:50.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T16:59:50.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:59:50.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T16:59:50.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:59:50.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:59:50.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:50.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:59:50.574 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:50.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:50.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:59:50.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:51.029 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:51.027+0000 7fa95c7cda40 -1 Falling back to public interface 2026-02-20T16:59:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:51.667 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:51.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:59:51.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:51.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:59:51.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:51.958 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:51.955+0000 7fa95c7cda40 -1 osd.0 0 log_to_monitors true 2026-02-20T16:59:52.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:52.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:52.760 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:59:52.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:59:52.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:52.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T16:59:52.917 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3707290820,v1:127.0.0.1:6801/3707290820] [v2:127.0.0.1:6802/3707290820,v1:127.0.0.1:6803/3707290820] exists,up 3fab5dce-12db-482a-b3b2-263f112efe1d 2026-02-20T16:59:52.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:59:52.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:59:52.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:52.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:59:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:59:52.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T16:59:52.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:59:52.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0419c833-7432-44ca-a924-b3b811011f4e 2026-02-20T16:59:52.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0419c833-7432-44ca-a924-b3b811011f4e' 2026-02-20T16:59:52.925 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 0419c833-7432-44ca-a924-b3b811011f4e 2026-02-20T16:59:52.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:59:52.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIk5hp4rqwNxAAzmCpYMVUOYXFV4vFVa8PSQ== 2026-02-20T16:59:52.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIk5hp4rqwNxAAzmCpYMVUOYXFV4vFVa8PSQ=="}' 2026-02-20T16:59:52.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0419c833-7432-44ca-a924-b3b811011f4e -i td/osd-scrub-repair/1/new.json 2026-02-20T16:59:53.090 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:53.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T16:59:53.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIk5hp4rqwNxAAzmCpYMVUOYXFV4vFVa8PSQ== --osd-uuid 0419c833-7432-44ca-a924-b3b811011f4e 2026-02-20T16:59:53.118 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.115+0000 7f091a474a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:53.119 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.115+0000 7f091a474a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:53.120 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.119+0000 7f091a474a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:53.121 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.119+0000 7f091a474a40 -1 bdev(0x55f87332cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T16:59:53.121 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.119+0000 7f091a474a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T16:59:53.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T16:59:53.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:59:53.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T16:59:53.772 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T16:59:53.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:59:53.976 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T16:59:53.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T16:59:53.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:59:53.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:59:53.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:59:53.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:59:54.000 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.999+0000 7f743a57ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:54.002 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.999+0000 7f743a57ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:54.003 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:53.999+0000 7f743a57ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:54.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:59:54.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:54.397 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:54.395+0000 7f743a57ca40 -1 Falling back to public interface 2026-02-20T16:59:55.017 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:55.015+0000 7f743a57ca40 -1 osd.1 0 log_to_monitors true 2026-02-20T16:59:55.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:55.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:55.297 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:55.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:59:55.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:55.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:59:55.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:56.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:56.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:56.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T16:59:56.462 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:59:56.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:56.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T16:59:56.623 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/371727609,v1:127.0.0.1:6811/371727609] [v2:127.0.0.1:6812/371727609,v1:127.0.0.1:6813/371727609] exists,up 0419c833-7432-44ca-a924-b3b811011f4e 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T16:59:56.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T16:59:56.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T16:59:56.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T16:59:56.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T16:59:56.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T16:59:56.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T16:59:56.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T16:59:56.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9385d08e-cdc2-427f-a821-c7593d7f64bc 2026-02-20T16:59:56.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9385d08e-cdc2-427f-a821-c7593d7f64bc' 2026-02-20T16:59:56.630 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 9385d08e-cdc2-427f-a821-c7593d7f64bc 2026-02-20T16:59:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T16:59:56.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMk5hp81M5JhAA2l0MFrr1Wx7ZJSHgaHOQSQ== 2026-02-20T16:59:56.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMk5hp81M5JhAA2l0MFrr1Wx7ZJSHgaHOQSQ=="}' 2026-02-20T16:59:56.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9385d08e-cdc2-427f-a821-c7593d7f64bc -i td/osd-scrub-repair/2/new.json 2026-02-20T16:59:56.806 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T16:59:56.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T16:59:56.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMk5hp81M5JhAA2l0MFrr1Wx7ZJSHgaHOQSQ== --osd-uuid 9385d08e-cdc2-427f-a821-c7593d7f64bc 2026-02-20T16:59:56.834 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:56.831+0000 7f2bd7897a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:56.835 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:56.831+0000 7f2bd7897a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:56.836 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:56.835+0000 7f2bd7897a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:56.836 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:56.835+0000 7f2bd7897a40 -1 bdev(0x55e14fab4c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T16:59:56.837 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:56.835+0000 7f2bd7897a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T16:59:57.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T16:59:57.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T16:59:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T16:59:57.499 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T16:59:57.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T16:59:57.698 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T16:59:57.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T16:59:57.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T16:59:57.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T16:59:57.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T16:59:57.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T16:59:57.722 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:57.719+0000 7f3dde3a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:57.723 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:57.719+0000 7f3dde3a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:57.724 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:57.723+0000 7f3dde3a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:57.856 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T16:59:57.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T16:59:57.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:57.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:59:58.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T16:59:58.117 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:58.115+0000 7f3dde3a7a40 -1 Falling back to public interface 2026-02-20T16:59:58.686 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T16:59:58.683+0000 7f3dde3a7a40 -1 osd.2 0 log_to_monitors true 2026-02-20T16:59:59.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T16:59:59.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T16:59:59.019 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T16:59:59.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T16:59:59.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T16:59:59.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T16:59:59.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:00.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:00.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:00.174 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:00.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:00.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:00.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:00.333 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/810179386,v1:127.0.0.1:6819/810179386] [v2:127.0.0.1:6820/810179386,v1:127.0.0.1:6821/810179386] exists,up 9385d08e-cdc2-427f-a821-c7593d7f64bc 2026-02-20T17:00:00.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:00.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:00.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:00.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T17:00:00.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T17:00:00.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:00.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:00.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T17:00:00.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:00.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9f567f14-aa89-4022-a81d-faf03b9e35ba 2026-02-20T17:00:00.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 9f567f14-aa89-4022-a81d-faf03b9e35ba' 2026-02-20T17:00:00.342 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 9f567f14-aa89-4022-a81d-faf03b9e35ba 2026-02-20T17:00:00.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:00.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQk5hpJmboFBAALsLmjjc+X7m8YzWX4j0WKg== 2026-02-20T17:00:00.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQk5hpJmboFBAALsLmjjc+X7m8YzWX4j0WKg=="}' 2026-02-20T17:00:00.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9f567f14-aa89-4022-a81d-faf03b9e35ba -i td/osd-scrub-repair/3/new.json 2026-02-20T17:00:00.512 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T17:00:00.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T17:00:00.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQk5hpJmboFBAALsLmjjc+X7m8YzWX4j0WKg== --osd-uuid 9f567f14-aa89-4022-a81d-faf03b9e35ba 2026-02-20T17:00:00.547 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:00.543+0000 7f3d8ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:00.549 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:00.547+0000 7f3d8ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:00.550 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:00.547+0000 7f3d8ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:00.550 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:00.547+0000 7f3d8ac13a40 -1 bdev(0x5586fef6cc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:00.550 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:00.547+0000 7f3d8ac13a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T17:00:01.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T17:00:01.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:01.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T17:00:01.175 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T17:00:01.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:01.380 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T17:00:01.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T17:00:01.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:01.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:01.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:01.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:01.397 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:01.395+0000 7fd8f03b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:01.399 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:01.395+0000 7fd8f03b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:01.399 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:01.395+0000 7fd8f03b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:01.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:01.539 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:01.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:01.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:01.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:01.793 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:01.791+0000 7fd8f03b1a40 -1 Falling back to public interface 2026-02-20T17:00:02.351 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:02.347+0000 7fd8f03b1a40 -1 osd.3 0 log_to_monitors true 2026-02-20T17:00:02.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:02.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:02.699 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:02.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:02.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:02.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:02.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:03.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:03.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:03.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:03.866 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:03.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:03.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1022107065,v1:127.0.0.1:6827/1022107065] [v2:127.0.0.1:6828/1022107065,v1:127.0.0.1:6829/1022107065] exists,up 9f567f14-aa89-4022-a81d-faf03b9e35ba 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-20T17:00:04.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T17:00:04.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-20T17:00:04.227 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T17:00:04.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T17:00:04.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T17:00:04.532 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T17:00:04.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:00:05.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-20T17:00:05.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T17:00:05.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:05.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:05.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:05.546 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:05.546 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:00:05.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:05.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:05.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:05.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:05.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:05.674 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:05.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:05.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T17:00:05.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T17:00:05.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T17:00:05.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:05.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:05.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T17:00:05.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T17:00:05.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-20T17:00:05.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:05.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:06.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-20T17:00:06.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-20T17:00:06.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542147' 2026-02-20T17:00:06.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:06.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:06.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411330 2026-02-20T17:00:06.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411330 2026-02-20T17:00:06.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542147 3-77309411330' 2026-02-20T17:00:06.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:06.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T17:00:06.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:06.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:06.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T17:00:06.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:06.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T17:00:06.131 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T17:00:06.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T17:00:06.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:06.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T17:00:06.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:07.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:07.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:07.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T17:00:07.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:08.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:00:08.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:08.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T17:00:08.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:08.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T17:00:08.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:08.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:08.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T17:00:08.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:08.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T17:00:08.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T17:00:08.628 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T17:00:08.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:08.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-20T17:00:08.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:08.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-20T17:00:08.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:08.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:08.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-20T17:00:08.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:08.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-20T17:00:08.791 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542147 2026-02-20T17:00:08.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-20T17:00:08.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:08.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-20T17:00:08.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:08.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411330 2026-02-20T17:00:08.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:08.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:08.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411330 2026-02-20T17:00:08.968 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:08.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411330 2026-02-20T17:00:08.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411330' 2026-02-20T17:00:08.970 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 77309411330 2026-02-20T17:00:08.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:09.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411330 -lt 77309411330 2026-02-20T17:00:09.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:09.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:09.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:09.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:09.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:09.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:09.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:09.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:09.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:09.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:09.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T17:00:09.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T17:00:09.884 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:00:09.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T17:00:10.089 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:00:10.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T17:00:10.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T17:00:10.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T17:00:10.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T17:00:10.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T17:00:10.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T17:00:10.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:10.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:00:10.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-20T17:00:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T17:00:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-20T17:00:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T17:00:10.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T17:00:10.292 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:10.292 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T17:00:10.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T17:00:10.449 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:00:10.449 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T17:00:10.449 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T17:00:10.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:10.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444584"' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444584' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444585"' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444585' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444588"' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444588' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 444584 444585 444588' 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T17:00:10.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444584 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444587: /' 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444590: /' 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:10.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444592: /' 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T17:00:11.108 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:11.109 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:11.112 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: start osd.0 2026-02-20T17:00:11.113 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:4444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T17:00:11.124 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:11.125 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:11.128 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: start osd.2 2026-02-20T17:00:11.129 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:4444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:11.156 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:11.157 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:11.159 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:11.159 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:11.159 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: start osd.3 2026-02-20T17:00:11.160 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2026-02-20T17:00:11.127+0000 7f78ec232a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2026-02-20T17:00:11.131+0000 7f78ec232a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.605 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2026-02-20T17:00:11.131+0000 7f78ec232a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: 0 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2026-02-20T17:00:11.631+0000 7f78ec232a40 -1 Falling back to public interface 2026-02-20T17:00:12.606 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2026-02-20T17:00:12.355+0000 7f78ec232a40 -1 osd.0 26 log_to_monitors true 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: 1 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: osd.0 up in weight 1 up_from 31 up_thru 0 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6818/3617192208,v1:127.0.0.1:6819/3617192208] [v2:127.0.0.1:6820/3617192208,v1:127.0.0.1:6821/3617192208] exists,up 3fab5dce-12db-482a-b3b2-263f112efe1d 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:12.607 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:12.608 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:12.608 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:12.608 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:12.608 INFO:tasks.workunit.client.0.trial013.stderr:444587: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444587: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get44590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2026-02-20T17:00:11.143+0000 7fb2d3651a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2026-02-20T17:00:11.147+0000 7fb2d3651a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.628 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2026-02-20T17:00:11.147+0000 7fb2d3651a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: 0 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:12.629 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2026-02-20T17:00:11.659+0000 7fb2d3651a40 -1 Falling back to public interface 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2026-02-20T17:00:12.315+0000 7fb2d3651a40 -1 osd.2 26 log_to_monitors true 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: 1 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: osd.2 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [14,26) [v2:127.0.0.1:6826/3711969299,v1:127.0.0.1:6827/3711969299] [v2:127.0.0.1:6828/3711969299,v1:127.0.0.1:6829/3711969299] exists,up 9385d08e-cdc2-427f-a821-c7593d7f64bc 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:12.630 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:12.631 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:12.631 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:12.631 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:12.631 INFO:tasks.workunit.client.0.trial013.stderr:444590: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:12.661 INFO:tasks.workunit.client.0.trial013.stderr:444590: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge44592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T17:00:12.661 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T17:00:12.661 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2026-02-20T17:00:11.171+0000 7f2369a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2026-02-20T17:00:11.171+0000 7f2369a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2026-02-20T17:00:11.171+0000 7f2369a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.662 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: 0 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2026-02-20T17:00:11.623+0000 7f2369a6fa40 -1 Falling back to public interface 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2026-02-20T17:00:12.279+0000 7f2369a6fa40 -1 osd.3 26 log_to_monitors true 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: 1 2026-02-20T17:00:12.663 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: osd.3 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [18,26) [v2:127.0.0.1:6800/2908230986,v1:127.0.0.1:6801/2908230986] [v2:127.0.0.1:6802/2908230986,v1:127.0.0.1:6803/2908230986] exists,up 9f567f14-aa89-4022-a81d-faf03b9e35ba 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:12.664 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:12.665 INFO:tasks.workunit.client.0.trial013.stderr:444592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:13.173 INFO:tasks.workunit.client.0.trial013.stderr:444592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge_timeout_delays: echo true 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:13.174 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: 1 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: 2 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: 3' 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178' 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-20T17:00:13.175 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-42949672966' 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-42949672966 2-128849018882' 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-42949672966 2-128849018882 3-128849018882' 2026-02-20T17:00:13.176 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:13.177 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986178 2026-02-20T17:00:13.213 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/t_timeout_delays: echo true 2026-02-20T17:00:13.213 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:13.213 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: 1 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: 2 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: 3' 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:13.214 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179' 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-42949672967' 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:13.215 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-42949672967 2-128849018883' 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-42949672967 2-128849018883 3-128849018883' 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:13.216 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986179 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444590: //homet_timeout_delays: echo true 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:13.249 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: 1 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: 2 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: 3' 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-20T17:00:13.250 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180' 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-42949672968' 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-42949672968 2-128849018884' 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:13.251 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:13.252 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-20T17:00:13.252 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-20T17:00:13.252 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-42949672968 2-128849018884 3-128849018884' 2026-02-20T17:00:13.252 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:13.252 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986180 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444592: //homeubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986178 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986178' 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: waiting osd.0 seq 133143986178 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986178 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:14.830 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: waiting osd.1 seq 42949672966 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672966 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018882 2026-02-20T17:00:14.831 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018882 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018882' 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: waiting osd.2 seq 128849018882 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-20T17:00:14.832 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986179 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986179 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986179' 2026-02-20T17:00:14.899 INFO:tasks.workunit.client.0.trial013.stderr:444590: waiting osd.0 seq 133143986179 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986179 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: waiting osd.1 seq 42949672967 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018883 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018883 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: waiting osd.2 seq 128849018883 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-20T17:00:14.900 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.906 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.906 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986180 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986180' 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: waiting osd.0 seq 133143986180 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986180 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: waiting osd.1 seq 42949672968 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018884 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018884 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018884' 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: waiting osd.2 seq 128849018884 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-20T17:00:14.907 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s:2267: flush_pg_stats: echo 3-128849018882 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018882 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-20T17:00:15.550 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018882' 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: waiting osd.3 seq 128849018882 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:15.551 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:444587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:15.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:15.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:15.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444585 2026-02-20T17:00:15.622 INFO:tasks.workunit.client.0.trial013.stderr:h:2267: flush_pg_stats: echo 3-128849018883 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018883 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018883' 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: waiting osd.3 seq 128849018883 2026-02-20T17:00:15.623 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:15.624 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:444590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:15.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:15.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:15.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444588 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:h:2267: flush_pg_stats: echo 3-128849018884 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018884 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018884' 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: waiting osd.3 seq 128849018884 2026-02-20T17:00:15.635 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:15.636 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:15.637 INFO:tasks.workunit.client.0.trial013.stderr:444592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:15.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:15.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T17:00:15.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-20T17:00:15.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-20T17:00:15.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T17:00:15.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T17:00:15.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:15.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:15.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:15.796 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:15.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:15.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T17:00:16.100 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-20T17:00:16.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:16.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:00:16.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:16.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:16.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:04.531995+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:16.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:17.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:17.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:17.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:17.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:17.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:17.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:17.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:17.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:04.531995+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:17.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:18.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:18.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:18.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:18.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:18.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:18.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:18.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:18.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:04.531995+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:18.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:19.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:19.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:19.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:19.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:19.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:19.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:19.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:19.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:04.531995+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:19.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:20.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:20.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:20.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:20.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:20.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:20.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:20.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:20.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:04.531995+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:20.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:21.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:21.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:21.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:21.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:21.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:21.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:21.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:17.013492+0000 '>' 2026-02-20T17:00:04.531995+0000 2026-02-20T17:00:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:00:22.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-20T17:00:22.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-20T17:00:22.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-20T17:00:22.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-20T17:00:22.272 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:22.272 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:22.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-20T17:00:22.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-20T17:00:22.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-20T17:00:22.463 INFO:tasks.workunit.client.0.trial013.stdout: osd: 4 osds: 4 up (since 9s), 4 in (since 21s) 2026-02-20T17:00:22.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-20T17:00:22.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-20T17:00:22.652 INFO:tasks.workunit.client.0.trial013.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 22s) 2026-02-20T17:00:22.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-20T17:00:22.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-20T17:00:22.845 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:22.846 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:22.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:22.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:22.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:22.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:22.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:22.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:00:22.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:00:22.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:00:22.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:00:22.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:00:22.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:00:22.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:22.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:00:22.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:00:22.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:22.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:00:22.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:00:22.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T17:00:22.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:00:22.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:22.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:22.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T17:00:22.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:00:22.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T17:00:22.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:22.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:22.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:22.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:22.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:22.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:22.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:00:22.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:00:22.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:00:22.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:00:22.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:00:22.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:00:22.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:22.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:00:22.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:00:22.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:22.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:00:22.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:00:22.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T17:00:22.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:00:22.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:22.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:22.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T17:00:22.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:00:22.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:00:22.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-20T17:00:22.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:00:22.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:22.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:22.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39930 2026-02-20T17:00:23.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:966: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=true 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-20T17:00:23.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-20T17:00:23.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-20T17:00:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:00:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:23.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:23.025 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:23.025 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:23.025 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:23.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:23.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T17:00:23.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:00:23.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:00:23.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:00:23.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:00:23.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:23.066 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:23.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T17:00:23.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:00:23.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get fsid 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:00:23.123 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:23.124 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39930/ceph-mon.a.asok 2026-02-20T17:00:23.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:00:23.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39930/ceph-mon.a.asok config get mon_host 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-20T17:00:23.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:00:23.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:00:23.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:23.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:23.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:23.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:23.277 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:23.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:23.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:00:23.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:00:23.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-20T17:00:23.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:23.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:23.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:23.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:23.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:23.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:23.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2974c754-3d60-4f9e-b089-859775a06d3b 2026-02-20T17:00:23.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2974c754-3d60-4f9e-b089-859775a06d3b' 2026-02-20T17:00:23.308 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 2974c754-3d60-4f9e-b089-859775a06d3b 2026-02-20T17:00:23.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAnk5hpkHDhEhAACer2Zy9dzyP6236KcwJwkg== 2026-02-20T17:00:23.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAnk5hpkHDhEhAACer2Zy9dzyP6236KcwJwkg=="}' 2026-02-20T17:00:23.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2974c754-3d60-4f9e-b089-859775a06d3b -i td/osd-scrub-repair/0/new.json 2026-02-20T17:00:23.406 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:23.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-20T17:00:23.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnk5hpkHDhEhAACer2Zy9dzyP6236KcwJwkg== --osd-uuid 2974c754-3d60-4f9e-b089-859775a06d3b 2026-02-20T17:00:23.431 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:23.426+0000 7fb9eb202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:23.432 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:23.430+0000 7fb9eb202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:23.433 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:23.430+0000 7fb9eb202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:23.434 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:23.430+0000 7fb9eb202a40 -1 bdev(0x55bc40cfec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:23.434 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:23.430+0000 7fb9eb202a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-20T17:00:24.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-20T17:00:24.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:24.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:00:24.128 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:00:24.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:24.222 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:00:24.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:00:24.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:24.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:24.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:24.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:24.238 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:24.234+0000 7f0f96dd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:24.239 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:24.234+0000 7f0f96dd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:24.240 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:24.238+0000 7f0f96dd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:24.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:24.318 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:24.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:24.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:24.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:24.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:24.634+0000 7f0f96dd6a40 -1 Falling back to public interface 2026-02-20T17:00:25.285 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:25.282+0000 7f0f96dd6a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:25.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:25.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:26.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:26.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:26.543 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:26.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:26.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:26.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:26.700 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1876367860,v1:127.0.0.1:6801/1876367860] [v2:127.0.0.1:6802/1876367860,v1:127.0.0.1:6803/1876367860] exists,up 2974c754-3d60-4f9e-b089-859775a06d3b 2026-02-20T17:00:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-20T17:00:26.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:26.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:26.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-20T17:00:26.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:26.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8afe0220-eda8-4faa-89d7-71b5d8224b5a 2026-02-20T17:00:26.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8afe0220-eda8-4faa-89d7-71b5d8224b5a' 2026-02-20T17:00:26.708 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 8afe0220-eda8-4faa-89d7-71b5d8224b5a 2026-02-20T17:00:26.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:26.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqk5hpbtHAKhAA3xMfYvdhmwQJHgZk5ZaTFw== 2026-02-20T17:00:26.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqk5hpbtHAKhAA3xMfYvdhmwQJHgZk5ZaTFw=="}' 2026-02-20T17:00:26.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8afe0220-eda8-4faa-89d7-71b5d8224b5a -i td/osd-scrub-repair/1/new.json 2026-02-20T17:00:26.878 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:26.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-20T17:00:26.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqk5hpbtHAKhAA3xMfYvdhmwQJHgZk5ZaTFw== --osd-uuid 8afe0220-eda8-4faa-89d7-71b5d8224b5a 2026-02-20T17:00:26.914 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:26.910+0000 7fbe06657a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:26.915 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:26.910+0000 7fbe06657a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:26.916 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:26.914+0000 7fbe06657a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:26.917 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:26.914+0000 7fbe06657a40 -1 bdev(0x561eda2d8c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:26.917 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:26.914+0000 7fbe06657a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-20T17:00:27.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-20T17:00:27.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:27.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:00:27.545 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:00:27.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:27.754 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:00:27.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:00:27.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:27.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:27.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:27.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:27.779 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:27.774+0000 7f0fb02b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:27.781 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:27.778+0000 7f0fb02b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:27.782 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:27.778+0000 7f0fb02b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:00:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:00:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:27.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:27.912 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:27.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:27.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:27.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:00:28.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:28.173 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:28.170+0000 7f0fb02b3a40 -1 Falling back to public interface 2026-02-20T17:00:28.851 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:28.846+0000 7f0fb02b3a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:00:29.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:29.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:29.074 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:29.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:29.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:29.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:00:29.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:30.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:30.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:30.231 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:30.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:30.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:30.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:00:30.393 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/364737770,v1:127.0.0.1:6811/364737770] [v2:127.0.0.1:6812/364737770,v1:127.0.0.1:6813/364737770] exists,up 8afe0220-eda8-4faa-89d7-71b5d8224b5a 2026-02-20T17:00:30.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:30.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:30.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:30.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T17:00:30.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:30.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c9a78e15-5c79-4f8c-8dad-1ac090c274d3 2026-02-20T17:00:30.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c9a78e15-5c79-4f8c-8dad-1ac090c274d3' 2026-02-20T17:00:30.401 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 c9a78e15-5c79-4f8c-8dad-1ac090c274d3 2026-02-20T17:00:30.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:30.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuk5hpkp10GBAAvazWIgiHinMId+Hk16iEvQ== 2026-02-20T17:00:30.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuk5hpkp10GBAAvazWIgiHinMId+Hk16iEvQ=="}' 2026-02-20T17:00:30.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c9a78e15-5c79-4f8c-8dad-1ac090c274d3 -i td/osd-scrub-repair/2/new.json 2026-02-20T17:00:30.574 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:30.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-20T17:00:30.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuk5hpkp10GBAAvazWIgiHinMId+Hk16iEvQ== --osd-uuid c9a78e15-5c79-4f8c-8dad-1ac090c274d3 2026-02-20T17:00:30.610 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:30.606+0000 7f49979d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:30.611 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:30.606+0000 7f49979d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:30.612 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:30.610+0000 7f49979d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:30.613 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:30.610+0000 7f49979d9a40 -1 bdev(0x556d9e756c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:30.613 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:30.610+0000 7f49979d9a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-20T17:00:31.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-20T17:00:31.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:31.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:00:31.233 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:00:31.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:31.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:31.464 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:31.462+0000 7fc58c7a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:31.466 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:31.462+0000 7fc58c7a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:31.467 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:31.462+0000 7fc58c7a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:00:31.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:31.606 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:31.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:31.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:31.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:31.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:31.886+0000 7fc58c7a0a40 -1 Falling back to public interface 2026-02-20T17:00:32.554 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:32.550+0000 7fc58c7a0a40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:00:32.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:32.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:32.771 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:32.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:32.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:32.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:32.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:33.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:33.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:33.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:33.938 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:33.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:33.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:34.098 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/642320975,v1:127.0.0.1:6819/642320975] [v2:127.0.0.1:6820/642320975,v1:127.0.0.1:6821/642320975] exists,up c9a78e15-5c79-4f8c-8dad-1ac090c274d3 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-20T17:00:34.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:34.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:34.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:34.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T17:00:34.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:34.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ae369cf5-1b44-4207-a51a-f0f71ef169a1 2026-02-20T17:00:34.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 ae369cf5-1b44-4207-a51a-f0f71ef169a1' 2026-02-20T17:00:34.107 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 ae369cf5-1b44-4207-a51a-f0f71ef169a1 2026-02-20T17:00:34.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:34.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyk5hpNn3mBhAA9sEU/ceADQ2vh3Q9jrQRbQ== 2026-02-20T17:00:34.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyk5hpNn3mBhAA9sEU/ceADQ2vh3Q9jrQRbQ=="}' 2026-02-20T17:00:34.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ae369cf5-1b44-4207-a51a-f0f71ef169a1 -i td/osd-scrub-repair/3/new.json 2026-02-20T17:00:34.278 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T17:00:34.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-20T17:00:34.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyk5hpNn3mBhAA9sEU/ceADQ2vh3Q9jrQRbQ== --osd-uuid ae369cf5-1b44-4207-a51a-f0f71ef169a1 2026-02-20T17:00:34.314 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:34.310+0000 7fdf7720ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:34.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:34.310+0000 7fdf7720ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:34.316 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:34.314+0000 7fdf7720ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:34.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:34.314+0000 7fdf7720ea40 -1 bdev(0x5627745fcc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:34.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:34.314+0000 7fdf7720ea40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-20T17:00:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-20T17:00:34.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:34.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-20T17:00:34.933 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T17:00:34.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:35.132 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T17:00:35.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-20T17:00:35.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:35.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:35.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:35.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:35.147 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:35.142+0000 7fadcaa23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:35.148 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:35.146+0000 7fadcaa23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:35.149 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:35.146+0000 7fadcaa23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:35.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:35.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:35.299 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:35.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:35.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:35.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:35.557 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:35.554+0000 7fadcaa23a40 -1 Falling back to public interface 2026-02-20T17:00:36.190 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:36.186+0000 7fadcaa23a40 -1 osd.3 0 log_to_monitors true 2026-02-20T17:00:36.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:36.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:36.455 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:36.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:36.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:36.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:36.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:37.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:37.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:37.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:00:37.620 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:00:37.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:37.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:37.775 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3419511341,v1:127.0.0.1:6827/3419511341] [v2:127.0.0.1:6828/3419511341,v1:127.0.0.1:6829/3419511341] exists,up ae369cf5-1b44-4207-a51a-f0f71ef169a1 2026-02-20T17:00:37.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-20T17:00:37.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-20T17:00:37.977 INFO:tasks.workunit.client.0.trial013.stderr:load: isa 2026-02-20T17:00:37.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-20T17:00:37.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-20T17:00:38.282 INFO:tasks.workunit.client.0.trial013.stderr:pool 'ecpool' created 2026-02-20T17:00:38.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:00:39.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-20T17:00:39.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-20T17:00:39.495 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 allow_ec_overwrites to true 2026-02-20T17:00:39.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-20T17:00:39.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:39.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:39.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:39.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:39.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:39.508 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:00:39.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:39.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:39.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:39.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:39.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:39.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:39.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:39.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-20T17:00:39.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-20T17:00:39.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-20T17:00:39.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:39.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:39.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T17:00:39.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T17:00:39.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-20T17:00:39.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:39.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:40.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-20T17:00:40.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-20T17:00:40.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542147' 2026-02-20T17:00:40.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:40.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:40.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378626 2026-02-20T17:00:40.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378626 2026-02-20T17:00:40.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542147 3-81604378626' 2026-02-20T17:00:40.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:40.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-20T17:00:40.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:40.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:40.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-20T17:00:40.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:40.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-20T17:00:40.099 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836484 2026-02-20T17:00:40.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-20T17:00:40.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:40.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T17:00:40.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:41.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:41.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:41.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-20T17:00:41.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:42.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:00:42.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:42.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-20T17:00:42.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:42.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T17:00:42.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:42.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:42.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T17:00:42.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:42.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T17:00:42.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T17:00:42.587 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T17:00:42.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:42.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-20T17:00:42.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:42.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-20T17:00:42.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:42.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:42.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-20T17:00:42.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:42.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-20T17:00:42.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-20T17:00:42.763 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542147 2026-02-20T17:00:42.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-20T17:00:42.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:42.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378626 2026-02-20T17:00:42.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:42.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:42.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378626 2026-02-20T17:00:42.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:42.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378626 2026-02-20T17:00:42.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378626' 2026-02-20T17:00:42.935 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 81604378626 2026-02-20T17:00:42.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:43.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378626 -lt 81604378626 2026-02-20T17:00:43.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:43.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:43.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:43.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:43.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:43.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:43.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:43.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:43.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:43.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:43.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:43.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:43.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:43.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-20T17:00:43.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-20T17:00:43.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-20T17:00:43.848 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:00:43.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-20T17:00:44.052 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:00:44.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-20T17:00:44.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-20T17:00:44.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-20T17:00:44.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-20T17:00:44.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-20T17:00:44.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T17:00:44.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:44.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:00:44.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-20T17:00:44.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-20T17:00:44.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-20T17:00:44.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-20T17:00:44.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-20T17:00:44.250 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:44.250 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-20T17:00:44.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-20T17:00:44.408 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:00:44.408 INFO:tasks.workunit.client.0.trial013.stderr:2 2026-02-20T17:00:44.408 INFO:tasks.workunit.client.0.trial013.stderr:3' 2026-02-20T17:00:44.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-20T17:00:44.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-20T17:00:44.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-20T17:00:44.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 455385"' 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 455385' 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:44.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 455387"' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 455387' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/455388: /' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 455389"' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 455389' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 455385 455387 455389' 2026-02-20T17:00:44.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 455385 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/455392: /' 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/455394: /' 2026-02-20T17:00:45.056 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:00:45.057 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:00:45.058 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:00:45.059 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:45.060 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-20T17:00:45.060 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:45.060 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:45.061 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:45.062 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: start osd.2 2026-02-20T17:00:45.063 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:45.072 INFO:tasks.workunit.client.0.trial013.stderr:4455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.072 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:45.072 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-20T17:00:45.072 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:45.073 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-20T17:00:45.074 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:00:45.075 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:45.076 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T17:00:45.077 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:00:45.078 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-20T17:00:45.078 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-20T17:00:45.078 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:45.078 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:45.079 INFO:tasks.workunit.client.0.trial013.stderr:455394: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:45.080 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455394: start osd.3 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:4activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:45.081 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:45.082 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: start osd.0 2026-02-20T17:00:45.083 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=dc862d09-5636-48d8-8909-4f9757603e12 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.39930/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:46.558 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-20T17:00:46.558 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:00:46.558 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2026-02-20T17:00:45.078+0000 7fc3c29f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2026-02-20T17:00:45.078+0000 7fc3c29f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2026-02-20T17:00:45.078+0000 7fc3c29f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:00:46.559 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: 0 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2026-02-20T17:00:45.562+0000 7fc3c29f7a40 -1 Falling back to public interface 2026-02-20T17:00:46.560 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2026-02-20T17:00:46.318+0000 7fc3c29f7a40 -1 osd.2 27 log_to_monitors true 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: 1 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: osd.2 up in weight 1 up_from 31 up_thru 0 down_at 28 last_clean_interval [14,27) [v2:127.0.0.1:6818/2850840740,v1:127.0.0.1:6819/2850840740] [v2:127.0.0.1:6820/2850840740,v1:127.0.0.1:6821/2850840740] exists,up c9a78e15-5c79-4f8c-8dad-1ac090c274d3 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:46.561 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:46.562 INFO:tasks.workunit.client.0.trial013.stderr:455392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge55394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2026-02-20T17:00:45.090+0000 7fe641170a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2026-02-20T17:00:45.090+0000 7fe641170a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2026-02-20T17:00:45.094+0000 7fe641170a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:46.563 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: 0 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2026-02-20T17:00:45.546+0000 7fe641170a40 -1 Falling back to public interface 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2026-02-20T17:00:46.294+0000 7fe641170a40 -1 osd.3 27 log_to_monitors true 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: 1 2026-02-20T17:00:46.564 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: osd.3 up in weight 1 up_from 31 up_thru 0 down_at 28 last_clean_interval [19,27) [v2:127.0.0.1:6800/4089064283,v1:127.0.0.1:6801/4089064283] [v2:127.0.0.1:6802/4089064283,v1:127.0.0.1:6803/4089064283] exists,up ae369cf5-1b44-4207-a51a-f0f71ef169a1 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:46.565 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:46.566 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:46.566 INFO:tasks.workunit.client.0.trial013.stderr:455394: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455394: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge55388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2026-02-20T17:00:45.090+0000 7f434020ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2026-02-20T17:00:45.090+0000 7f434020ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.580 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2026-02-20T17:00:45.094+0000 7f434020ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: 0 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:46.581 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2026-02-20T17:00:45.606+0000 7f434020ba40 -1 Falling back to public interface 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2026-02-20T17:00:46.386+0000 7f434020ba40 -1 osd.0 27 log_to_monitors true 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: 1 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: osd.0 up in weight 1 up_from 32 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6826/388246634,v1:127.0.0.1:6827/388246634] [v2:127.0.0.1:6828/388246634,v1:127.0.0.1:6829/388246634] exists,up 2974c754-3d60-4f9e-b089-859775a06d3b 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:00:46.582 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:00:46.583 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:00:46.583 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:00:46.583 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:00:46.583 INFO:tasks.workunit.client.0.trial013.stderr:455388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:00:47.132 INFO:tasks.workunit.client.0.trial013.stderr:455388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timt_timeout_delays: echo true 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:47.133 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: 1 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: 2 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: 3' 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953475 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953475 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953475' 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:47.134 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953475 1-42949672967' 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953475 1-42949672967 2-133143986178' 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-20T17:00:47.135 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953475 1-42949672967 2-133143986178 3-133143986178' 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-137438953475 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455394: //homet_timeout_delays: echo true 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:47.136 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: 1 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: 2 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: 3' 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:47.137 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953474 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953474 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953474' 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953474 1-42949672966' 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-20T17:00:47.138 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953474 1-42949672966 2-133143986179' 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953474 1-42949672966 2-133143986179 3-133143986179' 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:47.139 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-137438953474 2026-02-20T17:00:47.150 INFO:tasks.workunit.client.0.trial013.stderr:455392: //homeeout_delays: echo true 2026-02-20T17:00:47.150 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: 1 2026-02-20T17:00:47.151 INFO:tasks.workunit.client.0.trial013.stderr:455388: 2 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: 3' 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953476' 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953476 1-42949672968' 2026-02-20T17:00:47.152 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953476 1-42949672968 2-133143986180' 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-137438953476 1-42949672968 2-133143986180 3-133143986180' 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:47.153 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-137438953476 2026-02-20T17:00:48.784 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubun/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-137438953475 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953475 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 137438953475' 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: waiting osd.0 seq 137438953475 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 137438953475 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953475 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: waiting osd.1 seq 42949672967 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986178 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986178 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986178' 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: waiting osd.2 seq 133143986178 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-20T17:00:48.785 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-137438953474 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953474 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: waiting osd.0 seq 137438953474 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 137438953474 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953474 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: waiting osd.1 seq 42949672966 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672966 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986179 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986179 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986179 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: waiting osd.2 seq 133143986179 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-20T17:00:48.803 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.stu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-137438953476 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 137438953476' 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: waiting osd.0 seq 137438953476 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 137438953476 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953476 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-20T17:00:48.814 INFO:tasks.workunit.client.0.trial013.stderr:455388: waiting osd.1 seq 42949672968 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986180 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986180 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986180' 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: waiting osd.2 seq 133143986180 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-20T17:00:48.815 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226h:2267: flush_pg_stats: echo 3-133143986178 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-133143986178 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-20T17:00:49.500 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 133143986178' 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: waiting osd.3 seq 133143986178 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:49.501 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:49.502 INFO:tasks.workunit.client.0.trial013.stderr:455394: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:49.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:49.524 INFO:tasks.workunit.client.0.trial013.stderr:h:2267: flush_pg_stats: echo 3-133143986179 2026-02-20T17:00:49.524 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:49.524 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-133143986179 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986179 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 133143986179' 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: waiting osd.3 seq 133143986179 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.525 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:49.526 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:49.527 INFO:tasks.workunit.client.0.trial013.stderr:455392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:49.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:7: flush_pg_stats: echo 3-133143986180 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-133143986180 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-20T17:00:49.536 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 133143986180' 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: waiting osd.3 seq 133143986180 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:00:49.537 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:455388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:00:49.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 455387 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 455389 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-20T17:00:49.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-20T17:00:49.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-20T17:00:49.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-20T17:00:49.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T17:00:49.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-20T17:00:49.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T17:00:49.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-20T17:00:49.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-20T17:00:49.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T17:00:49.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T17:00:49.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:49.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:49.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:49.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:49.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T17:00:50.012 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-20T17:00:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:00:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:50.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:50.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:50.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:50.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:38.281546+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:50.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:51.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:51.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:51.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:51.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:51.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:51.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:51.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:51.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:38.281546+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:51.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:52.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:52.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:52.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:52.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:52.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:52.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:52.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:52.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:38.281546+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:52.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:53.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:53.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:53.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:53.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:53.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:53.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:53.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:53.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:38.281546+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:53.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:54.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:54.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:54.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:54.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:54.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:54.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:54.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:54.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:38.281546+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:54.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:00:55.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:00:55.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:00:55.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:00:55.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:00:55.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:00:55.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:00:55.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:00:56.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:00:51.037034+0000 '>' 2026-02-20T17:00:38.281546+0000 2026-02-20T17:00:56.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:00:56.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-20T17:00:56.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-20T17:00:56.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-20T17:00:56.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-20T17:00:56.192 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:56.193 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:56.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-20T17:00:56.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-20T17:00:56.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-20T17:00:56.387 INFO:tasks.workunit.client.0.trial013.stdout: osd: 4 osds: 4 up (since 9s), 4 in (since 22s) 2026-02-20T17:00:56.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-20T17:00:56.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-20T17:00:56.585 INFO:tasks.workunit.client.0.trial013.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 22s) 2026-02-20T17:00:56.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-20T17:00:56.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-20T17:00:56.782 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:56.782 INFO:tasks.workunit.client.0.trial013.stdout: 1/1 objects unfound (100.000%) 2026-02-20T17:00:56.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:56.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:57.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:57.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:00:57.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:00:57.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:00:57.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:00:57.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:00:57.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:00:57.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:00:57.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:00:57.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:00:57.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:00:57.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T17:00:57.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:00:57.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:57.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-repair 0 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-20T17:00:57.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T17:00:57.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-20T17:00:57.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:57.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:57.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:57.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:57.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:57.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:57.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:00:57.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:00:57.128 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:00:57.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:00:57.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:00:57.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:00:57.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:00:57.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:00:57.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:00:57.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T17:00:57.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-20T17:00:57.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:00:57.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39930 2026-02-20T17:00:57.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39930 2026-02-20T17:00:57.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:00:57.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:00:57.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T17:00:57.140 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T17:00:57.141 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T17:00:57.149 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2026-02-20T17:00:57.150 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2026-02-20T17:00:57.205 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T17:00:57.208 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-snaps 2026-02-20T17:00:57.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2026-02-20T17:00:57.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2026-02-20T17:00:57.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2026-02-20T17:00:57.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2026-02-20T17:00:57.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2026-02-20T17:00:57.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none ' 2026-02-20T17:00:57.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2026-02-20T17:00:57.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T17:00:57.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2026-02-20T17:00:57.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T17:00:57.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_snaps_primary 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_snaps_replica' 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:00:57.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:00:57.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:00:57.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:00:57.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:00:57.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:00:57.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:00:57.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:00:57.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:00:57.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:00:57.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:00:57.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:00:57.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:00:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:00:57.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:00:57.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:00:57.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:00:57.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:00:57.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:00:57.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-20T17:00:57.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:00:57.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.459686 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2026-02-20T17:00:57.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.459686 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-20T17:00:57.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2026-02-20T17:00:57.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:00:57.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2026-02-20T17:00:57.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:00:57.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:00:57.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:00:57.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:00:57.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:00:57.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:00:57.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get fsid 2026-02-20T17:00:57.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:00:57.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:00:57.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:00:57.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get mon_host 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:00:57.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-20T17:00:57.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:00:57.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:00:57.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:00:57.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:00:57.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:00:57.503 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2026-02-20T17:00:57.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:00:57.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:00:57.509 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:00:57.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:00:57.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:00:57.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:00:57.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:00:57.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=639b1570-f54c-4347-b155-6089bc996217 2026-02-20T17:00:57.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 639b1570-f54c-4347-b155-6089bc996217' 2026-02-20T17:00:57.512 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 639b1570-f54c-4347-b155-6089bc996217 2026-02-20T17:00:57.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:00:57.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJk5hpLIfxHhAAA+cligjo1yXs5RdIc5a5/w== 2026-02-20T17:00:57.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJk5hpLIfxHhAAA+cligjo1yXs5RdIc5a5/w=="}' 2026-02-20T17:00:57.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 639b1570-f54c-4347-b155-6089bc996217 -i td/osd-scrub-snaps/0/new.json 2026-02-20T17:00:57.610 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:57.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-20T17:00:57.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJk5hpLIfxHhAAA+cligjo1yXs5RdIc5a5/w== --osd-uuid 639b1570-f54c-4347-b155-6089bc996217 2026-02-20T17:00:57.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:57.630+0000 7f8fff7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:57.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:57.634+0000 7f8fff7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:57.638 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:57.634+0000 7f8fff7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:57.638 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:57.634+0000 7f8fff7dda40 -1 bdev(0x5627ba48ec00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:00:57.638 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:57.634+0000 7f8fff7dda40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-20T17:00:58.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-20T17:00:58.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:00:58.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:00:58.400 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:00:58.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:00:58.502 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:00:58.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:00:58.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:00:58.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:00:58.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:00:58.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:00:58.526 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:58.522+0000 7f7375a50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:58.528 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:58.526+0000 7f7375a50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:58.530 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:58.526+0000 7f7375a50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:00:58.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:00:58.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:00:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:00:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:00:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:00:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:58.594 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:00:58.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:00:58.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:58.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:58.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:00:58.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:58.918+0000 7f7375a50a40 -1 Falling back to public interface 2026-02-20T17:00:59.565 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:00:59.562+0000 7f7375a50a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:00:59.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:00:59.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:00:59.683 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:00:59.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:00:59.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:00:59.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:00:59.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:00.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:00.930 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2564415721,v1:127.0.0.1:6801/2564415721] [v2:127.0.0.1:6802/2564415721,v1:127.0.0.1:6803/2564415721] exists,up 639b1570-f54c-4347-b155-6089bc996217 2026-02-20T17:01:00.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:01:00.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:01:00.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:01:00.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2026-02-20T17:01:01.121 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:01:01.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2026-02-20T17:01:01.322 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:01:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2026-02-20T17:01:01.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:01:01.525 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:01:01.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:01:02.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2026-02-20T17:01:02.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:01:02.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:01:02.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:01:02.537 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:01:02.537 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:01:02.538 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:01:02.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:01:02.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:01:02.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:01:02.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:01:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T17:01:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:01:02.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:01:02.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:01:02.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-20T17:01:02.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-20T17:01:02.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-20T17:01:02.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:01:02.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-20T17:01:02.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:01:02.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:01:02.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-20T17:01:02.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:01:02.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-20T17:01:02.867 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836482 2026-02-20T17:01:02.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-20T17:01:02.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:03.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-20T17:01:03.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:01:04.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:01:04.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:04.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-20T17:01:04.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:01:04.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:04.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:04.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:01:04.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:01:04.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:01:04.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:01:04.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:01:04.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:04.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:04.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:01:04.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:01:04.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:01:04.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2026-02-20T17:01:04.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2026-02-20T17:01:04.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2026-02-20T17:01:04.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2026-02-20T17:01:04.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.459686 bs=1032 count=1 2026-02-20T17:01:04.900 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:01:04.900 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:01:04.900 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.6488e-05 s, 28.3 MB/s 2026-02-20T17:01:04.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2026-02-20T17:01:04.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:04.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.459686 2026-02-20T17:01:04.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:04.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.459686 2026-02-20T17:01:04.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:04.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.459686 2026-02-20T17:01:04.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:04.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.459686 2026-02-20T17:01:04.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:04.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.459686 2026-02-20T17:01:05.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.459686 2026-02-20T17:01:05.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.459686 2026-02-20T17:01:05.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.459686 2026-02-20T17:01:05.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.459686 2026-02-20T17:01:05.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.459686 2026-02-20T17:01:05.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.459686 2026-02-20T17:01:05.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.459686 2026-02-20T17:01:05.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.459686 2026-02-20T17:01:05.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.459686 2026-02-20T17:01:05.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.459686 2026-02-20T17:01:05.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-20T17:01:05.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.459686 2026-02-20T17:01:05.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2026-02-20T17:01:05.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:01:05.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:01:05.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:01:05.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.459686 0 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.459686 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-20T17:01:05.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-20T17:01:05.496 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap1 2026-02-20T17:01:05.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=1 2026-02-20T17:01:05.499 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:01:05.499 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:01:05.499 INFO:tasks.workunit.client.0.trial013.stderr:256 bytes copied, 8.598e-05 s, 3.0 MB/s 2026-02-20T17:01:05.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.459686 2026-02-20T17:01:05.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:05.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:01:05.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-20T17:01:05.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.459686 2026-02-20T17:01:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.459686 2026-02-20T17:01:05.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.459686 2026-02-20T17:01:05.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.459686 2026-02-20T17:01:05.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.459686 2026-02-20T17:01:05.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.459686 2026-02-20T17:01:05.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.459686 2026-02-20T17:01:05.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.459686 2026-02-20T17:01:05.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:05.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.459686 2026-02-20T17:01:05.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-20T17:01:05.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-20T17:01:05.853 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap2 2026-02-20T17:01:05.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=2 2026-02-20T17:01:05.856 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records in 2026-02-20T17:01:05.856 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records out 2026-02-20T17:01:05.856 INFO:tasks.workunit.client.0.trial013.stderr:512 bytes copied, 0.00030074 s, 1.7 MB/s 2026-02-20T17:01:05.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-20T17:01:05.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-20T17:01:05.954 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap3 2026-02-20T17:01:05.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=3 2026-02-20T17:01:05.956 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records in 2026-02-20T17:01:05.956 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records out 2026-02-20T17:01:05.957 INFO:tasks.workunit.client.0.trial013.stderr:768 bytes copied, 0.000193029 s, 4.0 MB/s 2026-02-20T17:01:05.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:01:05.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-20T17:01:05.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-20T17:01:06.056 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap4 2026-02-20T17:01:06.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=4 2026-02-20T17:01:06.059 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records in 2026-02-20T17:01:06.059 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records out 2026-02-20T17:01:06.060 INFO:tasks.workunit.client.0.trial013.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000177741 s, 5.8 MB/s 2026-02-20T17:01:06.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:06.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.459686 2026-02-20T17:01:06.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-20T17:01:06.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-20T17:01:06.159 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap5 2026-02-20T17:01:06.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-20T17:01:06.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-20T17:01:06.261 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap6 2026-02-20T17:01:06.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=6 2026-02-20T17:01:06.264 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records in 2026-02-20T17:01:06.265 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records out 2026-02-20T17:01:06.265 INFO:tasks.workunit.client.0.trial013.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000103132 s, 14.9 MB/s 2026-02-20T17:01:06.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:06.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-20T17:01:06.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-20T17:01:06.363 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap7 2026-02-20T17:01:06.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-20T17:01:06.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-20T17:01:06.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-20T17:01:06.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-20T17:01:06.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:01:06.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:01:06.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:01:06.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:01:06.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:01:06.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:01:06.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-20T17:01:06.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:06.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-20T17:01:07.131 INFO:tasks.workunit.client.0.trial013.stdout:WARNING: only removing head with clones present 2026-02-20T17:01:07.132 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T17:01:07.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:01:07.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-20T17:01:07.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:07.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:07.597 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:2# 2026-02-20T17:01:07.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:01:07.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-20T17:01:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:07.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:01:08.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:08.100 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.100 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.101 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.102 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.103 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.104 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.104 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:08.104 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:08.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:08.271 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:1# 2026-02-20T17:01:08.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:01:08.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:08.580 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.580 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:08.580 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.580 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.581 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:08.582 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:08.583 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:08.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:08.584 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:08.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:01:08.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:01:08.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-20T17:01:08.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:08.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=18 2026-02-20T17:01:08.756 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records in 2026-02-20T17:01:08.757 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records out 2026-02-20T17:01:08.757 INFO:tasks.workunit.client.0.trial013.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 9.1671e-05 s, 50.3 MB/s 2026-02-20T17:01:08.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:09.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-20T17:01:09.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:09.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=15 2026-02-20T17:01:09.200 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records in 2026-02-20T17:01:09.200 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records out 2026-02-20T17:01:09.200 INFO:tasks.workunit.client.0.trial013.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000149288 s, 25.7 MB/s 2026-02-20T17:01:09.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:09.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-20T17:01:09.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-20T17:01:09.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:09.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:09.863 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-20T17:01:09.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:01:10.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:10.179 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:10.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:10.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-20T17:01:10.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-20T17:01:10.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:10.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:10.514 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-20T17:01:10.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:01:10.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:10.832 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:10.832 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:10.833 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.834 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:10.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:10.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:01:10.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-20T17:01:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:11.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-20T17:01:11.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:11.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-20T17:01:11.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:11.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=7 2026-02-20T17:01:11.299 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records in 2026-02-20T17:01:11.300 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records out 2026-02-20T17:01:11.300 INFO:tasks.workunit.client.0.trial013.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 7.8556e-05 s, 22.8 MB/s 2026-02-20T17:01:11.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:11.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-20T17:01:11.699 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:11.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:11.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-20T17:01:12.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-20T17:01:12.186 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:12.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:12.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-20T17:01:12.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-20T17:01:12.683 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:12.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:12.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-20T17:01:13.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-20T17:01:13.118 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:13.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:13.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-20T17:01:13.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-20T17:01:13.562 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:13.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:13.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-20T17:01:13.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-20T17:01:14.056 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:14.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:14.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-20T17:01:14.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-20T17:01:14.518 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:14.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:14.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-20T17:01:14.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-20T17:01:14.983 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:15.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:15.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-20T17:01:15.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-20T17:01:15.403 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:15.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:15.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-20T17:01:15.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-20T17:01:15.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-20T17:01:15.879 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:15.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:15.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-20T17:01:16.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-20T17:01:16.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-20T17:01:16.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.459686 2026-02-20T17:01:16.278 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2026-02-20T17:01:16.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2026-02-20T17:01:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:01:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2026-02-20T17:01:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:01:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:01:16.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:01:16.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:01:16.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:01:16.286 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:01:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T17:01:16.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:01:16.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-20T17:01:16.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T17:01:16.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:01:16.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:01:16.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:01:16.302 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:16.297+0000 7f4fd8308a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:16.304 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:16.301+0000 7f4fd8308a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:16.304 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:16.301+0000 7f4fd8308a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:01:16.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:16.448 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:01:16.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:01:16.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:16.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:16.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:16.717 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:16.713+0000 7f4fd8308a40 -1 Falling back to public interface 2026-02-20T17:01:17.407 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:17.401+0000 7f4fd8308a40 -1 osd.0 17 log_to_monitors true 2026-02-20T17:01:17.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:17.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:17.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:01:17.612 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:01:17.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:17.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:17.772 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [5,17) [v2:127.0.0.1:6800/193488146,v1:127.0.0.1:6801/193488146] [v2:127.0.0.1:6802/193488146,v1:127.0.0.1:6803/193488146] exists,up 639b1570-f54c-4347-b155-6089bc996217 2026-02-20T17:01:17.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:01:17.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:01:17.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:01:17.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2026-02-20T17:01:17.836 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:17.837 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T17:01:17.837 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:17.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2026-02-20T17:01:17.923 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:17.924 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-20T17:01:17.924 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:17.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-20T17:01:18.003 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:18.003 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-20T17:01:18.004 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:18.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-20T17:01:18.086 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:18.086 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-20T17:01:18.087 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:18.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2026-02-20T17:01:18.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:01:18.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:01:18.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:01:18.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:01:18.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:01:18.096 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:01:18.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:01:18.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:01:18.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:01:18.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:01:18.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:01:18.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:01:18.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:01:18.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:01:18.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:01:18.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T17:01:18.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:01:18.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:01:18.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:01:18.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345922 2026-02-20T17:01:18.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345922 2026-02-20T17:01:18.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-85899345922' 2026-02-20T17:01:18.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:01:18.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-85899345922 2026-02-20T17:01:18.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:01:18.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:01:18.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-85899345922 2026-02-20T17:01:18.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:01:18.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345922 2026-02-20T17:01:18.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 85899345922' 2026-02-20T17:01:18.448 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 85899345922 2026-02-20T17:01:18.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:18.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 85899345922 2026-02-20T17:01:18.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:01:19.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:01:19.613 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:19.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345922 2026-02-20T17:01:19.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:01:19.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:19.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:19.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:01:19.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:01:19.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:01:19.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:01:19.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:01:19.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:01:19.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:01:19.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:01:20.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:01:20.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:01:20.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:20.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:01:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:01:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:01:20.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2026-02-20T17:01:20.391 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.391 INFO:tasks.workunit.client.0.trial013.stdout: "osd_shallow_scrub_chunk_max": "25" 2026-02-20T17:01:20.391 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2026-02-20T17:01:20.466 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.466 INFO:tasks.workunit.client.0.trial013.stdout: "osd_shallow_scrub_chunk_min": "5" 2026-02-20T17:01:20.466 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2026-02-20T17:01:20.540 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.540 INFO:tasks.workunit.client.0.trial013.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2026-02-20T17:01:20.540 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2026-02-20T17:01:20.614 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.614 INFO:tasks.workunit.client.0.trial013.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2026-02-20T17:01:20.614 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2026-02-20T17:01:20.689 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.689 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_chunk_max": "15" 2026-02-20T17:01:20.690 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2026-02-20T17:01:20.760 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:01:20.760 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_chunk_min": "5" 2026-02-20T17:01:20.760 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:01:20.767 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:01:20.768 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:01:20.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:01:20.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:01:20.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:01:21.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:01:21.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:01:21.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:01:21.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:01:21.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:01:21.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:01:21.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-20T17:01:21.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:01:21.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:01:21.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:01:21.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345923 2026-02-20T17:01:21.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345923 2026-02-20T17:01:21.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-85899345923' 2026-02-20T17:01:21.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:01:21.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-85899345923 2026-02-20T17:01:21.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:01:21.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:01:21.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-85899345923 2026-02-20T17:01:21.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:01:21.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345923 2026-02-20T17:01:21.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 85899345923' 2026-02-20T17:01:21.262 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 85899345923 2026-02-20T17:01:21.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:21.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345923 2026-02-20T17:01:21.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:01:22.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:01:22.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:22.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345923 -lt 85899345923 2026-02-20T17:01:22.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:01:22.583 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:01:22.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:01:22.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:01:22.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:01:22.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:01:22.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:01:22.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:01:22.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:22.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:22.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:22.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-20T17:01:22.958 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-20T17:01:22.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:22.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:22.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:22.970 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:23.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:01.522934+0000 '>' 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:23.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:01:24.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:01:24.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:01:24.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:01:24.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:24.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:24.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:24.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:24.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:01.522934+0000 '>' 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:24.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:01:25.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:01:25.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:01:25.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:01:25.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:25.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:25.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:25.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:25.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:01.522934+0000 '>' 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:25.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:01:26.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:01:26.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:01:26.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:01:26.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:26.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:26.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:26.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:26.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:01.522934+0000 '>' 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:26.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:01:27.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:01:27.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:01:27.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:01:27.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:01:27.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:01:27.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:01:27.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:01:27.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:23.429972+0000 '>' 2026-02-20T17:01:01.522934+0000 2026-02-20T17:01:27.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:01:27.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:27.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2026-02-20T17:01:27.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2026-02-20T17:01:27.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2026-02-20T17:01:27.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2026-02-20T17:01:27.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2026-02-20T17:01:27.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-20T17:01:27.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2026-02-20T17:01:27.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2026-02-20T17:01:27.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2026-02-20T17:01:27.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:01:27.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-20T17:01:27.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:01:27.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:01:27.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:01:27.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2026-02-20T17:01:27.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2026-02-20T17:01:27.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:01:27.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-20T17:01:27.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:01:27.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:01:27.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:01:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2026-02-20T17:01:27.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2026-02-20T17:01:27.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2026-02-20T17:01:27.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2026-02-20T17:01:27.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2026-02-20T17:01:27.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2026-02-20T17:01:27.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2026-02-20T17:01:27.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2026-02-20T17:01:27.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='475118 ' 2026-02-20T17:01:27.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2026-02-20T17:01:27.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2026-02-20T17:01:27.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:27.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2026-02-20T17:01:27.985 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap1 2026-02-20T17:01:27.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:27.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2026-02-20T17:01:28.086 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap2 2026-02-20T17:01:28.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:28.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2026-02-20T17:01:28.189 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap3 2026-02-20T17:01:28.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:28.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2026-02-20T17:01:28.290 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap4 2026-02-20T17:01:28.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:28.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2026-02-20T17:01:28.392 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap5 2026-02-20T17:01:28.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:28.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2026-02-20T17:01:28.495 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap6 2026-02-20T17:01:28.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-20T17:01:28.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2026-02-20T17:01:28.596 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap7 2026-02-20T17:01:28.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2026-02-20T17:01:33.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2026-02-20T17:01:33.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2026-02-20T17:01:33.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2026-02-20T17:01:33.747 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:01:33.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2026-02-20T17:01:33.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2026-02-20T17:01:33.906 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:01:33.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2026-02-20T17:01:33.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2026-02-20T17:01:34.065 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T17:01:34.065 INFO:tasks.workunit.client.0.trial013.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2026-02-20T17:01:28.595587+0000 23'60 29:141 [0] 0 [0] 0 17'56 2026-02-20T17:01:23.429972+0000 0'0 2026-02-20T17:01:01.522934+0000 7 0 periodic scrub scheduled @ 2026-02-21T18:42:42.798570+0000 33 0 2026-02-20T17:01:34.065 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:01:34.065 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T17:01:34.066 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:01:34.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2026-02-20T17:01:34.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 475118 2026-02-20T17:01:34.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2026-02-20T17:01:34.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:01:34.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:01:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:01:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:01:34.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2026-02-20T17:01:39.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2026-02-20T17:01:39.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2026-02-20T17:01:39.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-20T17:01:39.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:01:39.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2026-02-20T17:01:39.438 INFO:tasks.workunit.client.0.trial013.stdout:TEST PASSED 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:01:39.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:01:39.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:01:39.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:01:39.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:01:39.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:01:39.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:01:39.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:01:39.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:01:39.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:01:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:01:39.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:01:39.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:01:39.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:01:39.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:01:39.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:01:39.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:01:39.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:01:39.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-20T17:01:39.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:01:39.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:01:39.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:01:39.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:01:39.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:01:39.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:01:39.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:01:39.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:01:39.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:01:39.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:01:39.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:01:39.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:01:39.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:01:39.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:01:39.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:01:39.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:01:39.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:01:39.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:01:39.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:01:39.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:01:39.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-20T17:01:39.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:01:39.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.459686 2026-02-20T17:01:39.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.459686 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-20T17:01:39.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-20T17:01:39.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:01:39.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:01:39.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:01:39.504 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:01:39.505 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.505 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:39.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T17:01:39.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:01:39.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:01:39.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:01:39.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:01:39.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:01:39.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:01:39.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get fsid 2026-02-20T17:01:39.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:01:39.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:01:39.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:01:39.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:01:39.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:01:39.597 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:01:39.598 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:01:39.598 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:01:39.598 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:01:39.599 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.599 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.599 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:01:39.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:01:39.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get mon_host 2026-02-20T17:01:39.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-20T17:01:39.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:01:39.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:39.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:01:39.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:01:39.771 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-20T17:01:39.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:01:39.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:39.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:01:39.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:01:39.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:01:39.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f388208e-8efc-48f3-a0d1-6f3fe76787fe 2026-02-20T17:01:39.777 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 f388208e-8efc-48f3-a0d1-6f3fe76787fe 2026-02-20T17:01:39.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f388208e-8efc-48f3-a0d1-6f3fe76787fe' 2026-02-20T17:01:39.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:01:39.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBzk5hpSWUvLxAAXlCA1mfq+yeaS4dugvZ0Lw== 2026-02-20T17:01:39.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBzk5hpSWUvLxAAXlCA1mfq+yeaS4dugvZ0Lw=="}' 2026-02-20T17:01:39.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f388208e-8efc-48f3-a0d1-6f3fe76787fe -i td/osd-scrub-snaps/0/new.json 2026-02-20T17:01:39.888 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:01:39.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-20T17:01:39.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzk5hpSWUvLxAAXlCA1mfq+yeaS4dugvZ0Lw== --osd-uuid f388208e-8efc-48f3-a0d1-6f3fe76787fe 2026-02-20T17:01:39.918 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:39.913+0000 7fc56da55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:39.920 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:39.917+0000 7fc56da55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:39.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:39.917+0000 7fc56da55a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:39.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:39.917+0000 7fc56da55a40 -1 bdev(0x564f7272ac00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:01:39.921 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:39.917+0000 7fc56da55a40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-20T17:01:40.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-20T17:01:40.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:01:40.825 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:01:40.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:01:40.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:01:40.928 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:01:40.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:01:40.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:01:40.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:01:40.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:01:40.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:01:40.949 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:40.945+0000 7f6164346a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:40.950 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:40.945+0000 7f6164346a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:40.951 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:40.945+0000 7f6164346a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:01:41.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:41.029 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:01:41.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:01:41.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:41.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:41.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:41.377 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:41.373+0000 7f6164346a40 -1 Falling back to public interface 2026-02-20T17:01:42.098 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:42.093+0000 7f6164346a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:01:42.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:42.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:42.127 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:01:42.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:01:42.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:42.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:42.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:43.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:43.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:43.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:01:43.289 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:01:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:01:43.458 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3877693919,v1:127.0.0.1:6801/3877693919] [v2:127.0.0.1:6802/3877693919,v1:127.0.0.1:6803/3877693919] exists,up f388208e-8efc-48f3-a0d1-6f3fe76787fe 2026-02-20T17:01:43.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:01:43.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:01:43.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:01:43.464 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:01:43.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-20T17:01:43.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:01:43.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f8b610cb-23f6-4a08-b134-e3ac0e2aa66a 2026-02-20T17:01:43.467 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 f8b610cb-23f6-4a08-b134-e3ac0e2aa66a 2026-02-20T17:01:43.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f8b610cb-23f6-4a08-b134-e3ac0e2aa66a' 2026-02-20T17:01:43.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:01:43.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB3k5hpHOt0HBAAcPnDMp6xJT0nX5kzGcKGNQ== 2026-02-20T17:01:43.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB3k5hpHOt0HBAAcPnDMp6xJT0nX5kzGcKGNQ=="}' 2026-02-20T17:01:43.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f8b610cb-23f6-4a08-b134-e3ac0e2aa66a -i td/osd-scrub-snaps/1/new.json 2026-02-20T17:01:43.640 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:01:43.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-20T17:01:43.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3k5hpHOt0HBAAcPnDMp6xJT0nX5kzGcKGNQ== --osd-uuid f8b610cb-23f6-4a08-b134-e3ac0e2aa66a 2026-02-20T17:01:43.670 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:43.665+0000 7f80cb32aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:43.671 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:43.665+0000 7f80cb32aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:43.672 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:43.669+0000 7f80cb32aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:43.673 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:43.669+0000 7f80cb32aa40 -1 bdev(0x5623505dec00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:01:43.673 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:43.669+0000 7f80cb32aa40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-20T17:01:44.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-20T17:01:44.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:01:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:01:44.404 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:01:44.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:01:44.604 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:01:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:01:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:01:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:01:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:01:44.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:01:44.633 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:44.629+0000 7f61ccda4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:44.635 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:44.629+0000 7f61ccda4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:44.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:44.633+0000 7f61ccda4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:01:44.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:44.771 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:01:44.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:01:44.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:44.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:01:44.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:45.065 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:45.061+0000 7f61ccda4a40 -1 Falling back to public interface 2026-02-20T17:01:45.665 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:01:45.661+0000 7f61ccda4a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:01:45.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:45.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:45.941 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:01:45.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:01:45.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:45.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:01:46.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:01:47.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:01:47.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:01:47.105 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:01:47.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:01:47.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:01:47.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:01:47.263 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/784060979,v1:127.0.0.1:6811/784060979] [v2:127.0.0.1:6812/784060979,v1:127.0.0.1:6813/784060979] exists,up f8b610cb-23f6-4a08-b134-e3ac0e2aa66a 2026-02-20T17:01:47.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:01:47.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:01:47.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:01:47.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-20T17:01:47.465 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:01:47.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-20T17:01:47.669 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:01:47.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-20T17:01:47.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:01:47.872 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:01:47.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:01:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-20T17:01:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:01:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:01:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:01:48.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:01:48.884 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:01:48.884 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:01:48.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:01:48.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:01:48.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:01:48.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:01:48.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:01:48.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:01:48.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:01:48.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:01:48.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:01:49.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:01:49.140 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:01:49.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:01:49.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:01:49.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:01:49.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T17:01:49.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T17:01:49.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T17:01:49.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:01:49.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:01:49.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-20T17:01:49.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-20T17:01:49.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-20T17:01:49.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:01:49.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T17:01:49.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:01:49.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:01:49.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T17:01:49.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:01:49.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T17:01:49.303 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T17:01:49.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T17:01:49.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:49.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T17:01:49.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:01:50.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:01:50.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:01:50.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T17:01:50.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:01:50.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-20T17:01:50.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:01:50.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:01:50.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-20T17:01:50.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:01:50.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-20T17:01:50.643 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672962 2026-02-20T17:01:50.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-20T17:01:50.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:01:50.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-20T17:01:50.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:01:50.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:50.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:51.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:01:51.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:01:51.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:01:51.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:01:51.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:01:51.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:01:51.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:01:51.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:01:51.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:01:51.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:01:51.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:01:51.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:01:51.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:01:51.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:01:51.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:01:51.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-20T17:01:51.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-20T17:01:51.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-20T17:01:51.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-20T17:01:51.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.459686 bs=1032 count=1 2026-02-20T17:01:51.544 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:01:51.544 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:01:51.544 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.6307e-05 s, 28.4 MB/s 2026-02-20T17:01:51.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-20T17:01:51.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.459686 2026-02-20T17:01:51.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.459686 2026-02-20T17:01:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.459686 2026-02-20T17:01:51.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.459686 2026-02-20T17:01:51.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.459686 2026-02-20T17:01:51.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.459686 2026-02-20T17:01:51.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.459686 2026-02-20T17:01:51.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.459686 2026-02-20T17:01:51.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.459686 2026-02-20T17:01:51.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.459686 2026-02-20T17:01:51.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.459686 2026-02-20T17:01:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.459686 2026-02-20T17:01:51.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.459686 2026-02-20T17:01:51.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.459686 2026-02-20T17:01:51.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.459686 2026-02-20T17:01:51.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:01:51.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.459686 2026-02-20T17:01:51.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-20T17:01:51.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:01:51.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:01:51.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:01:51.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:01:52.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:01:52.151 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:01:52.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T17:01:52.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-20T17:01:52.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T17:01:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-20T17:01:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.459686 '$primary' 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.459686 1 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.459686 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-20T17:01:52.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-20T17:01:52.556 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap1 2026-02-20T17:01:52.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=1 2026-02-20T17:01:52.559 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:01:52.559 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:01:52.559 INFO:tasks.workunit.client.0.trial013.stderr:256 bytes copied, 8.644e-05 s, 3.0 MB/s 2026-02-20T17:01:52.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.459686 2026-02-20T17:01:52.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:52.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:01:52.646 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-20T17:01:52.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.459686 2026-02-20T17:01:52.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.459686 2026-02-20T17:01:52.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.459686 2026-02-20T17:01:52.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.459686 2026-02-20T17:01:52.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.459686 2026-02-20T17:01:52.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.459686 2026-02-20T17:01:52.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.459686 2026-02-20T17:01:52.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.459686 2026-02-20T17:01:52.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:01:52.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.459686 2026-02-20T17:01:52.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-20T17:01:52.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-20T17:01:52.925 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap2 2026-02-20T17:01:52.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=2 2026-02-20T17:01:52.929 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records in 2026-02-20T17:01:52.929 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records out 2026-02-20T17:01:52.929 INFO:tasks.workunit.client.0.trial013.stderr:512 bytes copied, 8.6441e-05 s, 5.9 MB/s 2026-02-20T17:01:52.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:52.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-20T17:01:52.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-20T17:01:53.027 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap3 2026-02-20T17:01:53.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=3 2026-02-20T17:01:53.031 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records in 2026-02-20T17:01:53.031 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records out 2026-02-20T17:01:53.031 INFO:tasks.workunit.client.0.trial013.stderr:768 bytes copied, 9.0468e-05 s, 8.5 MB/s 2026-02-20T17:01:53.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:01:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-20T17:01:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-20T17:01:53.130 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap4 2026-02-20T17:01:53.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=4 2026-02-20T17:01:53.132 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records in 2026-02-20T17:01:53.132 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records out 2026-02-20T17:01:53.132 INFO:tasks.workunit.client.0.trial013.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.8112e-05 s, 10.4 MB/s 2026-02-20T17:01:53.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:53.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.459686 2026-02-20T17:01:53.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-20T17:01:53.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-20T17:01:53.232 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap5 2026-02-20T17:01:53.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-20T17:01:53.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-20T17:01:53.334 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap6 2026-02-20T17:01:53.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=6 2026-02-20T17:01:53.336 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records in 2026-02-20T17:01:53.337 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records out 2026-02-20T17:01:53.337 INFO:tasks.workunit.client.0.trial013.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 9.8583e-05 s, 15.6 MB/s 2026-02-20T17:01:53.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:01:53.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-20T17:01:53.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-20T17:01:53.435 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap7 2026-02-20T17:01:53.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-20T17:01:53.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-20T17:01:53.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-20T17:01:53.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-20T17:01:53.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:01:53.519 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:01:53.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:01:53.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:01:53.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:01:53.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:01:53.827 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2026-02-20T17:01:54.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:54.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-20T17:01:54.215 INFO:tasks.workunit.client.0.trial013.stdout:WARNING: only removing head with clones present 2026-02-20T17:01:54.216 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T17:01:54.321 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-20T17:01:54.321 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-20T17:01:54.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:54.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:54.682 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:2# 2026-02-20T17:01:54.781 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-20T17:01:54.781 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-20T17:01:54.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:54.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:54.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-20T17:01:55.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:55.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.180 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:55.181 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.182 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.183 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.184 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:55.184 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:55.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:55.386 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:1# 2026-02-20T17:01:55.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-20T17:01:55.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:55.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:55.697 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:55.698 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:55.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:55.699 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:55.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:01:55.699 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-20T17:01:55.699 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-20T17:01:55.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:55.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=18 2026-02-20T17:01:55.872 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records in 2026-02-20T17:01:55.873 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records out 2026-02-20T17:01:55.873 INFO:tasks.workunit.client.0.trial013.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 9.7271e-05 s, 47.4 MB/s 2026-02-20T17:01:55.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:56.148 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2026-02-20T17:01:56.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:56.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=15 2026-02-20T17:01:56.312 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records in 2026-02-20T17:01:56.312 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records out 2026-02-20T17:01:56.313 INFO:tasks.workunit.client.0.trial013.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000328392 s, 11.7 MB/s 2026-02-20T17:01:56.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:56.621 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2026-02-20T17:01:56.621 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-20T17:01:56.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:56.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:57.015 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-20T17:01:57.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-20T17:01:57.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:57.357 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:57.358 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:57.359 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:57.360 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.360 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:57.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:57.360 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.360 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2026-02-20T17:01:57.361 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-20T17:01:57.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:57.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:01:57.685 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-20T17:01:57.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-20T17:01:58.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:01:58.004 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:01:58.004 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:01:58.004 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:58.005 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:58.006 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:01:58.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:01:58.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:01:58.007 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2026-02-20T17:01:58.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:58.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-20T17:01:58.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:58.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-20T17:01:58.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:58.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=7 2026-02-20T17:01:58.502 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records in 2026-02-20T17:01:58.502 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records out 2026-02-20T17:01:58.502 INFO:tasks.workunit.client.0.trial013.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000204741 s, 8.8 MB/s 2026-02-20T17:01:58.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:01:58.796 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2026-02-20T17:01:58.870 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:58.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:58.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-20T17:01:59.253 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2026-02-20T17:01:59.334 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:59.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:59.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-20T17:01:59.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2026-02-20T17:01:59.770 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:01:59.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:01:59.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-20T17:02:00.180 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2026-02-20T17:02:00.263 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:00.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:00.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-20T17:02:00.677 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2026-02-20T17:02:00.750 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:00.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:00.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-20T17:02:01.128 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2026-02-20T17:02:01.207 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:01.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:01.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-20T17:02:01.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2026-02-20T17:02:01.658 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:01.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:01.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-20T17:02:02.020 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2026-02-20T17:02:02.106 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:02.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:02.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-20T17:02:02.505 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2026-02-20T17:02:02.590 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:02.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:02.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-20T17:02:02.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-20T17:02:02.964 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2026-02-20T17:02:03.022 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:03.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:03.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-20T17:02:03.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-20T17:02:03.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-20T17:02:03.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.459686 2026-02-20T17:02:03.416 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-20T17:02:03.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:03.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:03.421 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:03.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:03.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:02:03.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:02:03.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:02:03.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T17:02:03.424 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:02:03.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:02:03.449 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:03.444+0000 7f5f8d0f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:03.450 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:03.444+0000 7f5f8d0f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:03.451 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:03.444+0000 7f5f8d0f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:03.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T17:02:03.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:03.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:02:03.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:03.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:03.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:03.591 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:03.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:03.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:03.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:03.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:03.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:03.884+0000 7f5f8d0f8a40 -1 Falling back to public interface 2026-02-20T17:02:04.602 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:04.596+0000 7f5f8d0f8a40 -1 osd.0 22 log_to_monitors true 2026-02-20T17:02:04.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:04.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:04.757 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:04.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:04.922 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/508653943,v1:127.0.0.1:6801/508653943] [v2:127.0.0.1:6802/508653943,v1:127.0.0.1:6803/508653943] exists,up f388208e-8efc-48f3-a0d1-6f3fe76787fe 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-20T17:02:04.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:04.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:04.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-20T17:02:04.928 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:02:04.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:02:04.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:02:04.953 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:04.948+0000 7f3927706a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:04.954 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:04.948+0000 7f3927706a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:04.955 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:04.948+0000 7f3927706a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:05.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:05.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:05.094 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:05.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:05.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:05.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:05.333 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:05.328+0000 7f3927706a40 -1 Falling back to public interface 2026-02-20T17:02:05.982 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:05.976+0000 7f3927706a40 -1 osd.1 22 log_to_monitors true 2026-02-20T17:02:06.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:06.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:06.255 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:06.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:06.418 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6810/1150427979,v1:127.0.0.1:6811/1150427979] [v2:127.0.0.1:6812/1150427979,v1:127.0.0.1:6813/1150427979] exists,up f8b610cb-23f6-4a08-b134-e3ac0e2aa66a 2026-02-20T17:02:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:06.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-20T17:02:06.488 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:06.488 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T17:02:06.488 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.496 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:06.496 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T17:02:06.496 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-20T17:02:06.584 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:06.584 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-20T17:02:06.584 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.591 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:06.591 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-20T17:02:06.591 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-20T17:02:06.676 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:06.676 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-20T17:02:06.676 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.683 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:06.683 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-20T17:02:06.683 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-20T17:02:06.769 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:06.770 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-20T17:02:06.770 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.776 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:06.776 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-20T17:02:06.776 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-20T17:02:06.863 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:06.863 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-20T17:02:06.863 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.870 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:06.870 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-20T17:02:06.870 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:06.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-20T17:02:06.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:02:06.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:02:06.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:02:06.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:02:06.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:02:06.881 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:02:06.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:02:06.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:02:06.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:02:06.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:02:06.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:02:06.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:02:06.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:02:06.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:02:06.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:02:07.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:02:07.143 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:02:07.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:02:07.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:07.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:02:07.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182402 2026-02-20T17:02:07.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182402 2026-02-20T17:02:07.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182402' 2026-02-20T17:02:07.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:07.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:02:07.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-20T17:02:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-20T17:02:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182402 1-120259084290' 2026-02-20T17:02:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-107374182402 2026-02-20T17:02:07.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:07.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:02:07.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-107374182402 2026-02-20T17:02:07.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:07.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182402 2026-02-20T17:02:07.302 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 107374182402 2026-02-20T17:02:07.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2026-02-20T17:02:07.302 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:07.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 107374182402 2026-02-20T17:02:07.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:08.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:02:08.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:08.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182402 -lt 107374182402 2026-02-20T17:02:08.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:08.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-20T17:02:08.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:08.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:02:08.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-20T17:02:08.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:08.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-20T17:02:08.644 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 120259084290 2026-02-20T17:02:08.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-20T17:02:08.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:02:08.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-20T17:02:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:02:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:08.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:09.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:02:09.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:02:09.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:02:09.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:02:09.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:02:09.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:02:09.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:02:09.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:02:09.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:02:09.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:02:09.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:09.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:09.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:02:09.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:02:09.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:02:09.375 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:02:09.376 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:02:09.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:02:09.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:02:09.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:02:09.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:02:09.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:02:09.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:02:09.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:02:09.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:02:09.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:02:09.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:02:09.801 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:02:09.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:02:09.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:09.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:02:09.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182404 2026-02-20T17:02:09.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182404 2026-02-20T17:02:09.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182404' 2026-02-20T17:02:09.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:09.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:02:09.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084291 2026-02-20T17:02:09.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084291 2026-02-20T17:02:09.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182404 1-120259084291' 2026-02-20T17:02:09.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:09.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-107374182404 2026-02-20T17:02:09.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:09.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:02:09.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-107374182404 2026-02-20T17:02:09.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:09.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182404 2026-02-20T17:02:09.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2026-02-20T17:02:09.959 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 107374182404 2026-02-20T17:02:09.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182404 -lt 107374182404 2026-02-20T17:02:10.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:10.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084291 2026-02-20T17:02:10.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:10.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:02:10.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084291 2026-02-20T17:02:10.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:10.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084291 2026-02-20T17:02:10.141 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 120259084291 2026-02-20T17:02:10.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084291' 2026-02-20T17:02:10.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:02:10.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084291 -lt 120259084291 2026-02-20T17:02:10.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:02:10.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:02:10.304 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:02:10.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:02:10.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:02:10.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:02:10.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:02:10.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-20T17:02:10.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:02:10.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:02:10.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:02:10.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:02:10.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T17:01:47.870035+0000 2026-02-20T17:02:10.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-20T17:02:10.701 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-20T17:02:10.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-20T17:01:47.870035+0000 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:01:47.870035+0000 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:02:10.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:02:10.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:02:10.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:01:47.870035+0000 '>' 2026-02-20T17:01:47.870035+0000 2026-02-20T17:02:10.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:02:11.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:02:11.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:02:11.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:02:11.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:02:11.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:02:11.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:02:11.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:02:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:02:10.969577+0000 '>' 2026-02-20T17:01:47.870035+0000 2026-02-20T17:02:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:02:12.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:12.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-20T17:02:12.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2026-02-20T17:02:12.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:02:12.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-20T17:02:12.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2026-02-20T17:02:12.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-20T17:02:12.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-20T17:02:12.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-20T17:02:12.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-20T17:02:12.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-20T17:02:12.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "epoch": 28, 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "inconsistents": [ 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj1", 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.121 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "version": 18 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "missing" 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj1", 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.122 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1828249343, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'18", 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'1", 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4222.0:1", 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 18, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.584423+0000", 2026-02-20T17:02:12.123 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.584776+0000", 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.124 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256 2026-02-20T17:02:12.125 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "missing" 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj10", 2026-02-20T17:02:12.126 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "version": 32 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj10", 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.127 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 718195851, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'32", 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'10", 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4243.0:1", 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 32, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.778289+0000", 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.778628+0000", 2026-02-20T17:02:12.128 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.129 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.130 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.131 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "????", 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.132 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj11", 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.133 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "version": 34 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj11", 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 693400951, 2026-02-20T17:02:12.134 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'34", 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'11", 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4246.0:1", 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 34, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.806937+0000", 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.807321+0000", 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.135 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.136 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.137 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.138 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.139 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj13", 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "version": 38 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.140 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj13", 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2087409765, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'38", 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'13", 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4252.0:1", 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 38, 2026-02-20T17:02:12.141 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.863742+0000", 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.864133+0000", 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.142 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.143 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.144 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.145 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]" 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj14", 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.146 INFO:tasks.workunit.client.0.trial013.stdout: "version": 40 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj14", 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2484217095, 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.147 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'40", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'14", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4255.0:1", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 40, 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.894098+0000", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.894490+0000", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.148 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.149 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.150 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.151 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1033, 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.152 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj15", 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "version": 15 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.153 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_corrupted" 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj15", 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 612772309, 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.154 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'15", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "0'0", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4207.0:1", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 15, 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:51.951661+0000", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:51.952107+0000", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.155 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x8a4d5e96", 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.156 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_corrupted" 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-20T17:02:12.157 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj2", 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "version": 56 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_missing" 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.158 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj2", 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1058988552, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "version": "22'56", 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "19'48", 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4297.0:1", 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 56, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0, 2026-02-20T17:02:12.159 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:53.514850+0000", 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:53.515173+0000", 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "whiteout", 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "dirty" 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xffffffff", 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.160 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 7, 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.161 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 4, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 4, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 3, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 2, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 7, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1024, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 7, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 6, 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: 5 2026-02-20T17:02:12.162 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_missing" 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj3", 2026-02-20T17:02:12.163 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "version": 44 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch" 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch_info", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "obj_size_info_mismatch" 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj3", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1643547569, 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.164 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "version": "18'44", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "16'22", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4267.0:1", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 44, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:53.058559+0000", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:53.058879+0000", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xae6ea20f", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.165 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.166 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch_info", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "obj_size_info_mismatch" 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "size": 3840, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "object_info": { 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj3", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1643547569, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "version": "18'44", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "16'22", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4267.0:1", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 44, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768, 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:53.058559+0000", 2026-02-20T17:02:12.167 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:53.058879+0000", 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xae6ea20f", 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.168 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj6", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "version": 24 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj6", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2202164420, 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.169 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'24", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'6", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4231.0:1", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 24, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.667729+0000", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.668226+0000", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.170 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.171 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj7", 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: "version": 26 2026-02-20T17:02:12.172 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj7", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1552453721, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'26", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'7", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4234.0:1", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 26, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.698952+0000", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.699413+0000", 2026-02-20T17:02:12.173 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.174 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.175 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj8", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "version": 28 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj8", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2381834917, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'28", 2026-02-20T17:02:12.176 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'8", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4237.0:1", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 28, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.719463+0000", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.719791+0000", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.177 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.178 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj9", 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "version": 30 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:02:12.179 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj9", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 3833113727, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "version": "16'30", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'9", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4240.0:1", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 30, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:01:52.748624+0000", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:01:52.749045+0000", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xe6d12a7e", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:02:12.180 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:02:12.181 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "size": "????", 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:02:12.182 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-20T17:02:12.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:02:12.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:02:12.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:02:12.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-20T17:02:12.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-20T17:02:12.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-20T17:02:12.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-20T17:02:12.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-20T17:02:12.189 INFO:tasks.workunit.client.0.trial013.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-20T17:02:12.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-20T17:02:12.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-20T17:02:12.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-20T17:02:12.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='494364 ' 2026-02-20T17:02:12.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-20T17:02:12.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-20T17:02:12.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='493822 ' 2026-02-20T17:02:12.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-20T17:02:12.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2026-02-20T17:02:12.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2026-02-20T17:02:12.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2026-02-20T17:02:12.266 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap1 2026-02-20T17:02:12.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2026-02-20T17:02:12.367 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap2 2026-02-20T17:02:12.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2026-02-20T17:02:12.469 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap3 2026-02-20T17:02:12.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2026-02-20T17:02:12.571 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap4 2026-02-20T17:02:12.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2026-02-20T17:02:12.674 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap5 2026-02-20T17:02:12.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2026-02-20T17:02:12.776 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap6 2026-02-20T17:02:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-20T17:02:12.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2026-02-20T17:02:12.877 INFO:tasks.workunit.client.0.trial013.stdout:removed pool test snap snap7 2026-02-20T17:02:12.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2026-02-20T17:02:17.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2026-02-20T17:02:17.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2026-02-20T17:02:17.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2026-02-20T17:02:18.032 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:02:18.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2026-02-20T17:02:18.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2026-02-20T17:02:18.213 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:02:18.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2026-02-20T17:02:18.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-20T17:02:18.389 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T17:02:18.389 INFO:tasks.workunit.client.0.trial013.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2026-02-20T17:02:12.876861+0000 22'56 37:134 [1,0] 1 [1,0] 1 22'56 2026-02-20T17:02:10.969577+0000 0'0 2026-02-20T17:01:47.870035+0000 7 1 periodic scrub scheduled @ 2026-02-21T21:17:34.671561+0000 33 0 2026-02-20T17:02:18.389 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:02:18.390 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T17:02:18.390 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:02:18.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-20T17:02:18.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 494364 2026-02-20T17:02:18.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-20T17:02:18.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 493822 2026-02-20T17:02:18.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-20T17:02:18.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:02:18.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:02:18.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:02:18.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:02:18.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-20T17:02:23.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-20T17:02:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-20T17:02:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-20T17:02:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2026-02-20T17:02:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:02:23.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-20T17:02:23.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2026-02-20T17:02:23.735 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2026-02-20T17:02:23.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2026-02-20T17:02:23.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-20T17:02:23.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-20T17:02:23.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:02:23.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-20T17:02:23.737 INFO:tasks.workunit.client.0.trial013.stdout:TEST PASSED 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:02:23.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:02:23.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:02:23.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:02:23.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:02:23.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:02:23.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:02:23.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:02:23.744 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:02:23.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:02:23.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:02:23.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:02:23.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:02:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:02:23.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:02:23.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:02:23.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:02:23.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:02:23.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:02:23.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:02:23.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:02:23.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:02:23.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:02:23.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:02:23.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:02:23.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:02:23.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:02:23.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:02:23.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:02:23.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:02:23.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:02:23.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:02:23.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:02:23.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:02:23.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:02:23.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:02:23.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:02:23.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:02:23.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:02:23.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:02:23.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:02:23.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:02:23.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:02:23.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:02:23.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:02:23.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-20T17:02:23.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:02:23.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.459686 2026-02-20T17:02:23.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.459686 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-20T17:02:23.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-20T17:02:23.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:02:23.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:23.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:23.807 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:23.807 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.807 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:23.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T17:02:23.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:02:23.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:02:23.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:02:23.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:02:23.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:02:23.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:02:23.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get fsid 2026-02-20T17:02:23.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:02:23.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:02:23.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:02:23.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:02:23.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.459686/ceph-mon.a.asok 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:02:23.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.459686/ceph-mon.a.asok config get mon_host 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-20T17:02:23.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:02:24.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:02:24.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:24.054 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:24.055 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:24.055 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:24.055 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:24.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:24.055 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:02:24.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:02:24.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-20T17:02:24.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-20T17:02:24.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:24.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:02:24.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:24.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:02:24.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:02:24.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 2026-02-20T17:02:24.087 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 2026-02-20T17:02:24.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3' 2026-02-20T17:02:24.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:02:24.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgk5hp6j/mBRAAKDseEcYQySbYeUcmFv1iGA== 2026-02-20T17:02:24.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgk5hp6j/mBRAAKDseEcYQySbYeUcmFv1iGA=="}' 2026-02-20T17:02:24.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 -i td/osd-scrub-snaps/0/new.json 2026-02-20T17:02:24.194 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:24.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-20T17:02:24.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgk5hp6j/mBRAAKDseEcYQySbYeUcmFv1iGA== --osd-uuid 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 2026-02-20T17:02:24.222 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:24.216+0000 7f12c1c57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:24.224 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:24.220+0000 7f12c1c57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:24.225 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:24.220+0000 7f12c1c57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:24.225 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:24.220+0000 7f12c1c57a40 -1 bdev(0x55ef3c908c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:02:24.225 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:24.220+0000 7f12c1c57a40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-20T17:02:24.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-20T17:02:24.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:02:25.000 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:02:25.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:02:25.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:02:25.109 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:02:25.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:02:25.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:25.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:02:25.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:02:25.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:02:25.140 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:25.136+0000 7f724a41ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:25.142 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:25.136+0000 7f724a41ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:25.143 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:25.136+0000 7f724a41ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:25.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:02:25.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:25.212 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:25.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:25.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:25.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:25.697 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:25.692+0000 7f724a41ea40 -1 Falling back to public interface 2026-02-20T17:02:26.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:26.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:26.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:26.312 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:26.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:26.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:26.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:26.603 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:26.596+0000 7f724a41ea40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:02:27.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:27.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:27.407 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:02:27.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:02:27.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:27.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:27.570 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2417569021,v1:127.0.0.1:6801/2417569021] [v2:127.0.0.1:6802/2417569021,v1:127.0.0.1:6803/2417569021] exists,up 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:02:27.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:02:27.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:27.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:02:27.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:27.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:02:27.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3642fabe-a216-40c2-8d63-0ce5ae5a014b 2026-02-20T17:02:27.577 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 3642fabe-a216-40c2-8d63-0ce5ae5a014b 2026-02-20T17:02:27.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3642fabe-a216-40c2-8d63-0ce5ae5a014b' 2026-02-20T17:02:27.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:02:27.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjk5hp6ywaIxAATQr9QukB6TzsoCMKqMD/iA== 2026-02-20T17:02:27.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjk5hp6ywaIxAATQr9QukB6TzsoCMKqMD/iA=="}' 2026-02-20T17:02:27.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3642fabe-a216-40c2-8d63-0ce5ae5a014b -i td/osd-scrub-snaps/1/new.json 2026-02-20T17:02:27.757 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:27.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-20T17:02:27.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjk5hp6ywaIxAATQr9QukB6TzsoCMKqMD/iA== --osd-uuid 3642fabe-a216-40c2-8d63-0ce5ae5a014b 2026-02-20T17:02:27.798 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:27.792+0000 7fe089c4ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:27.800 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:27.796+0000 7fe089c4ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:27.801 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:27.796+0000 7fe089c4ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:27.801 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:27.796+0000 7fe089c4ba40 -1 bdev(0x55f203082c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:02:27.801 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:27.796+0000 7fe089c4ba40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-20T17:02:28.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-20T17:02:28.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:02:28.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:02:28.413 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:02:28.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:02:28.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:02:28.625 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:02:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:02:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:02:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:02:28.657 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:28.652+0000 7f29920f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:28.659 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:28.652+0000 7f29920f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:28.661 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:28.656+0000 7f29920f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:28.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:28.789 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:28.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:28.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:28.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:29.029 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:29.024+0000 7f29920f0a40 -1 Falling back to public interface 2026-02-20T17:02:29.649 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:29.644+0000 7f29920f0a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:02:29.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:29.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:29.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:29.957 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:29.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:29.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:30.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:31.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:31.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:02:31.123 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:02:31.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:31.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:31.283 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3487293048,v1:127.0.0.1:6811/3487293048] [v2:127.0.0.1:6812/3487293048,v1:127.0.0.1:6813/3487293048] exists,up 3642fabe-a216-40c2-8d63-0ce5ae5a014b 2026-02-20T17:02:31.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:31.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:31.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:31.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-20T17:02:31.485 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:02:31.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-20T17:02:31.689 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:02:31.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-20T17:02:31.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:02:31.892 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:02:31.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:02:32.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-20T17:02:32.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:02:32.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:02:32.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:02:32.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:02:32.907 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:02:32.907 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:02:32.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:02:32.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:02:32.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:02:33.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:02:33.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:02:33.178 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:02:33.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:02:33.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:33.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:02:33.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T17:02:33.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T17:02:33.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T17:02:33.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:33.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:02:33.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-20T17:02:33.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-20T17:02:33.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-20T17:02:33.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:33.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T17:02:33.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:33.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:02:33.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T17:02:33.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:33.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T17:02:33.339 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T17:02:33.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T17:02:33.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:33.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-20T17:02:33.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:34.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:02:34.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:34.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-20T17:02:34.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:34.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-20T17:02:34.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:34.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:02:34.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-20T17:02:34.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:34.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-20T17:02:34.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-20T17:02:34.683 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705666 2026-02-20T17:02:34.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:02:34.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-20T17:02:34.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:02:34.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:34.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:02:35.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:02:35.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:02:35.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:02:35.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:02:35.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:02:35.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:35.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:35.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:02:35.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T17:02:35.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T17:02:35.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T17:02:35.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:02:35.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T17:02:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:02:35.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:02:35.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:02:35.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:02:35.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:02:35.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:35.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:35.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:02:35.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T17:02:35.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T17:02:35.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T17:02:35.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T17:02:35.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T17:02:35.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T17:02:35.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T17:02:35.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T17:02:36.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T17:02:36.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T17:02:36.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T17:02:36.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:02:36.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T17:02:36.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:02:36.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:02:36.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:02:36.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:02:36.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:02:36.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:02:36.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:02:36.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:02:36.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:02:36.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:02:36.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:36.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:36.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:02:36.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:02:36.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:02:36.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-20T17:02:36.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-20T17:02:36.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-20T17:02:36.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-20T17:02:36.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.459686 bs=1032 count=1 2026-02-20T17:02:36.798 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:02:36.798 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:02:36.799 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.6498e-05 s, 28.3 MB/s 2026-02-20T17:02:36.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-20T17:02:36.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.459686 2026-02-20T17:02:36.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.459686 2026-02-20T17:02:36.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.459686 2026-02-20T17:02:36.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.459686 2026-02-20T17:02:36.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.459686 2026-02-20T17:02:36.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.459686 2026-02-20T17:02:36.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.459686 2026-02-20T17:02:36.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:36.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.459686 2026-02-20T17:02:37.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.459686 2026-02-20T17:02:37.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.459686 2026-02-20T17:02:37.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.459686 2026-02-20T17:02:37.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.459686 2026-02-20T17:02:37.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.459686 2026-02-20T17:02:37.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.459686 2026-02-20T17:02:37.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.459686 2026-02-20T17:02:37.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-20T17:02:37.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.459686 2026-02-20T17:02:37.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-20T17:02:37.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:02:37.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:02:37.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:02:37.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:02:37.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:02:37.408 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:02:37.409 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:02:37.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T17:02:37.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-20T17:02:37.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T17:02:37.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-20T17:02:37.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.459686 '$replica' 2026-02-20T17:02:37.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.459686 0 2026-02-20T17:02:37.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-20T17:02:37.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-20T17:02:37.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.459686 2026-02-20T17:02:37.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-20T17:02:37.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-20T17:02:37.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-20T17:02:37.811 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap1 2026-02-20T17:02:37.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=1 2026-02-20T17:02:37.814 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:02:37.814 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:02:37.814 INFO:tasks.workunit.client.0.trial013.stderr:256 bytes copied, 6.0202e-05 s, 4.3 MB/s 2026-02-20T17:02:37.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.459686 2026-02-20T17:02:37.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:02:37.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:02:37.903 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-20T17:02:37.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:37.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.459686 2026-02-20T17:02:37.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:37.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.459686 2026-02-20T17:02:37.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:37.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.459686 2026-02-20T17:02:37.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:37.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.459686 2026-02-20T17:02:38.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:38.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.459686 2026-02-20T17:02:38.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:38.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.459686 2026-02-20T17:02:38.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:38.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.459686 2026-02-20T17:02:38.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:38.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.459686 2026-02-20T17:02:38.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-20T17:02:38.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.459686 2026-02-20T17:02:38.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-20T17:02:38.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-20T17:02:38.220 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap2 2026-02-20T17:02:38.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=2 2026-02-20T17:02:38.223 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records in 2026-02-20T17:02:38.223 INFO:tasks.workunit.client.0.trial013.stderr:2+0 records out 2026-02-20T17:02:38.223 INFO:tasks.workunit.client.0.trial013.stderr:512 bytes copied, 9.2672e-05 s, 5.5 MB/s 2026-02-20T17:02:38.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:02:38.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-20T17:02:38.253 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-20T17:02:38.321 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap3 2026-02-20T17:02:38.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=3 2026-02-20T17:02:38.324 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records in 2026-02-20T17:02:38.324 INFO:tasks.workunit.client.0.trial013.stderr:3+0 records out 2026-02-20T17:02:38.324 INFO:tasks.workunit.client.0.trial013.stderr:768 bytes copied, 8.9537e-05 s, 8.6 MB/s 2026-02-20T17:02:38.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.459686 2026-02-20T17:02:38.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-20T17:02:38.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-20T17:02:38.423 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap4 2026-02-20T17:02:38.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=4 2026-02-20T17:02:38.426 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records in 2026-02-20T17:02:38.426 INFO:tasks.workunit.client.0.trial013.stderr:4+0 records out 2026-02-20T17:02:38.426 INFO:tasks.workunit.client.0.trial013.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 8.9607e-05 s, 11.4 MB/s 2026-02-20T17:02:38.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:02:38.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.459686 2026-02-20T17:02:38.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-20T17:02:38.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-20T17:02:38.525 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap5 2026-02-20T17:02:38.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-20T17:02:38.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-20T17:02:38.627 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap6 2026-02-20T17:02:38.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=6 2026-02-20T17:02:38.630 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records in 2026-02-20T17:02:38.631 INFO:tasks.workunit.client.0.trial013.stderr:6+0 records out 2026-02-20T17:02:38.631 INFO:tasks.workunit.client.0.trial013.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 9.9886e-05 s, 15.4 MB/s 2026-02-20T17:02:38.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.459686 2026-02-20T17:02:38.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-20T17:02:38.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-20T17:02:38.728 INFO:tasks.workunit.client.0.trial013.stdout:created pool test snap snap7 2026-02-20T17:02:38.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-20T17:02:38.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-20T17:02:38.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-20T17:02:38.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-20T17:02:38.800 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:02:38.800 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:02:38.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:02:38.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:02:38.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:02:39.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:02:39.110 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-20T17:02:39.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:39.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-20T17:02:39.516 INFO:tasks.workunit.client.0.trial013.stdout:WARNING: only removing head with clones present 2026-02-20T17:02:39.516 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-20T17:02:39.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:02:39.625 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-20T17:02:39.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:39.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:02:39.982 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:2# 2026-02-20T17:02:40.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:02:40.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-20T17:02:40.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:40.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:40.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:02:40.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:02:40.532 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.532 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:02:40.532 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:02:40.533 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.534 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:40.535 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:40.536 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:40.536 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:40.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:02:40.537 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:40.537 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:02:40.725 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:c52c9666:::obj5:1# 2026-02-20T17:02:40.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:02:41.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:02:41.068 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.068 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:41.069 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:41.070 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:41.071 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:41.072 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-20T17:02:41.072 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:41.072 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:41.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:02:41.072 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:41.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:02:41.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-20T17:02:41.073 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-20T17:02:41.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:41.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=18 2026-02-20T17:02:41.261 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records in 2026-02-20T17:02:41.261 INFO:tasks.workunit.client.0.trial013.stderr:18+0 records out 2026-02-20T17:02:41.261 INFO:tasks.workunit.client.0.trial013.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 9.5418e-05 s, 48.3 MB/s 2026-02-20T17:02:41.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:02:41.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-20T17:02:41.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:41.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=15 2026-02-20T17:02:41.688 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records in 2026-02-20T17:02:41.688 INFO:tasks.workunit.client.0.trial013.stderr:15+0 records out 2026-02-20T17:02:41.688 INFO:tasks.workunit.client.0.trial013.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000146022 s, 26.3 MB/s 2026-02-20T17:02:41.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:02:41.972 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-20T17:02:41.972 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-20T17:02:42.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:42.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:02:42.363 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-20T17:02:42.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:02:42.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:02:42.692 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:02:42.692 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.692 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:42.692 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:02:42.693 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:42.694 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:42.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.696 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-20T17:02:42.697 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-20T17:02:42.697 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-20T17:02:42.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:42.869 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-20T17:02:43.079 INFO:tasks.workunit.client.0.trial013.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-20T17:02:43.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-20T17:02:43.403 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-20T17:02:43.404 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-20T17:02:43.405 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-20T17:02:43.406 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:43.407 INFO:tasks.workunit.client.0.trial013.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-20T17:02:43.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-20T17:02:43.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-20T17:02:43.407 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-20T17:02:43.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:43.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-20T17:02:43.869 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:43.869 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-20T17:02:43.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:43.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.459686 bs=256 count=7 2026-02-20T17:02:43.871 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records in 2026-02-20T17:02:43.871 INFO:tasks.workunit.client.0.trial013.stderr:7+0 records out 2026-02-20T17:02:43.872 INFO:tasks.workunit.client.0.trial013.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 7.6573e-05 s, 23.4 MB/s 2026-02-20T17:02:43.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.459686 2026-02-20T17:02:44.160 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-20T17:02:44.239 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:44.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:44.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-20T17:02:44.628 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-20T17:02:44.703 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:44.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:44.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-20T17:02:45.080 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-20T17:02:45.162 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:45.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:45.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-20T17:02:45.544 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-20T17:02:45.626 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:45.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:45.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-20T17:02:45.994 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-20T17:02:46.079 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:46.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:46.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-20T17:02:46.472 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-20T17:02:46.567 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:46.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:46.672 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-20T17:02:46.924 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-20T17:02:47.018 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:47.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:47.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-20T17:02:47.408 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-20T17:02:47.479 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:47.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:47.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-20T17:02:47.868 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-20T17:02:47.950 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:48.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:48.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-20T17:02:48.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-20T17:02:48.324 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-20T17:02:48.410 INFO:tasks.workunit.client.0.trial013.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-20T17:02:48.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:02:48.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-20T17:02:48.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-20T17:02:48.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-20T17:02:48.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.459686 2026-02-20T17:02:48.759 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-20T17:02:48.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:48.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:48.764 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:48.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:48.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:48.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:48.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:02:48.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:02:48.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-20T17:02:48.767 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:02:48.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-20T17:02:48.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:48.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-20T17:02:48.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-20T17:02:48.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:02:48.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:02:48.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:02:48.790 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:48.787+0000 7f2ddc250a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:48.792 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:48.791+0000 7f2ddc250a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:48.793 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:48.791+0000 7f2ddc250a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:48.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:48.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:48.933 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:48.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:48.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:49.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:49.182 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:49.179+0000 7f2ddc250a40 -1 Falling back to public interface 2026-02-20T17:02:49.802 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:49.799+0000 7f2ddc250a40 -1 osd.0 21 log_to_monitors true 2026-02-20T17:02:50.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:50.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:50.097 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:50.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:50.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:50.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:02:50.252 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/84564333,v1:127.0.0.1:6801/84564333] [v2:127.0.0.1:6802/84564333,v1:127.0.0.1:6803/84564333] exists,up 2bb0eaa8-916a-4951-9c75-b2dea13ca4a3 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:02:50.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:02:50.254 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:02:50.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-20T17:02:50.256 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:02:50.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T17:02:50.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.459686/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:02:50.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-20T17:02:50.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T17:02:50.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:02:50.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:02:50.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:02:50.279 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:50.275+0000 7f0af9ec4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:50.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:50.279+0000 7f0af9ec4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:50.281 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:50.279+0000 7f0af9ec4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:50.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:02:50.428 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:02:50.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:50.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:50.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:02:50.661 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:50.659+0000 7f0af9ec4a40 -1 Falling back to public interface 2026-02-20T17:02:51.377 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:02:51.375+0000 7f0af9ec4a40 -1 osd.1 21 log_to_monitors true 2026-02-20T17:02:51.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:02:51.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:02:51.591 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:02:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:02:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:02:51.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:02:51.748 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 22 last_clean_interval [9,21) [v2:127.0.0.1:6810/730197398,v1:127.0.0.1:6811/730197398] [v2:127.0.0.1:6812/730197398,v1:127.0.0.1:6813/730197398] exists,up 3642fabe-a216-40c2-8d63-0ce5ae5a014b 2026-02-20T17:02:51.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:02:51.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:02:51.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:02:51.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-20T17:02:51.817 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:51.818 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T17:02:51.818 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:51.825 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:51.825 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T17:02:51.825 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:51.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-20T17:02:51.904 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:51.904 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-20T17:02:51.904 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:51.911 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:51.911 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-20T17:02:51.911 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:51.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-20T17:02:51.997 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:51.998 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-20T17:02:51.998 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.005 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:52.005 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-20T17:02:52.005 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-20T17:02:52.085 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:52.085 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-20T17:02:52.085 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.092 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:52.093 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-20T17:02:52.093 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-20T17:02:52.173 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:02:52.173 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-20T17:02:52.173 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.180 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:02:52.180 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-20T17:02:52.180 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:02:52.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-20T17:02:52.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:02:52.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:02:52.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:02:52.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:02:52.189 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:02:52.189 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:02:52.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:02:52.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:02:52.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:02:52.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:02:52.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:02:52.465 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:02:52.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:02:52.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:52.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:02:52.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-20T17:02:52.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-20T17:02:52.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-20T17:02:52.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:52.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:02:52.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-20T17:02:52.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-20T17:02:52.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-120259084290' 2026-02-20T17:02:52.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:52.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-20T17:02:52.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:52.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:02:52.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-20T17:02:52.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:52.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-20T17:02:52.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-20T17:02:52.624 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 103079215106 2026-02-20T17:02:52.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:52.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-20T17:02:52.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:53.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:02:53.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:53.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-20T17:02:53.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:54.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:02:54.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:55.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-20T17:02:55.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:55.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-20T17:02:55.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:55.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:02:55.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-20T17:02:55.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:55.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-20T17:02:55.126 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 120259084290 2026-02-20T17:02:55.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-20T17:02:55.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:02:55.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-20T17:02:55.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:02:55.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:55.300 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:55.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:02:55.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:02:55.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:02:55.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:02:55.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:02:55.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:02:55.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:02:55.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:02:55.860 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:02:55.860 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:02:55.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:02:55.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:02:55.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:02:56.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:02:56.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:02:56.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:02:56.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:02:56.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:02:56.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:02:56.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:02:56.299 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:02:56.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:02:56.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:56.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:02:56.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-20T17:02:56.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-20T17:02:56.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-20T17:02:56.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:02:56.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:02:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084291 2026-02-20T17:02:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084291 2026-02-20T17:02:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-120259084291' 2026-02-20T17:02:56.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:56.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-20T17:02:56.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:56.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:02:56.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-20T17:02:56.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:56.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-20T17:02:56.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-20T17:02:56.456 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 103079215108 2026-02-20T17:02:56.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:56.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-20T17:02:56.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:57.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:02:57.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:57.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-20T17:02:57.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:02:58.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:02:58.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:02:58.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-20T17:02:58.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:02:58.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084291 2026-02-20T17:02:58.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:02:58.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:02:58.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084291 2026-02-20T17:02:58.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:02:58.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084291 2026-02-20T17:02:58.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084291' 2026-02-20T17:02:58.969 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 120259084291 2026-02-20T17:02:58.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:02:59.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084291 2026-02-20T17:02:59.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:02:59.137 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:02:59.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:02:59.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:02:59.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:02:59.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:02:59.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:02:59.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-20T17:02:59.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:02:59.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:02:59.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:02:59.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:02:59.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T17:02:31.890359+0000 2026-02-20T17:02:59.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-20T17:02:59.532 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-20T17:02:31.890359+0000 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:02:31.890359+0000 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:02:59.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:02:59.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:02:59.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:02:59.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:02:59.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:02:59.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:02:59.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:02:31.890359+0000 '>' 2026-02-20T17:02:31.890359+0000 2026-02-20T17:02:59.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:03:00.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:03:00.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:03:00.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:03:00.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:03:00.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:03:00.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:03:00.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:03:00.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:02:31.890359+0000 '>' 2026-02-20T17:02:31.890359+0000 2026-02-20T17:03:00.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:03:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:03:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:03:01.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:03:01.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:03:01.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:03:01.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:03:01.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:03:02.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:02:31.890359+0000 '>' 2026-02-20T17:02:31.890359+0000 2026-02-20T17:03:02.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:03:03.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:03:03.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-20T17:03:03.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:03:03.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:03:03.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:03:03.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:03:03.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:03:03.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:03:00.300156+0000 '>' 2026-02-20T17:02:31.890359+0000 2026-02-20T17:03:03.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:03:03.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:03.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-20T17:03:03.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2026-02-20T17:03:03.205 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:03:03.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-20T17:03:03.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2026-02-20T17:03:03.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-20T17:03:03.235 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-20T17:03:03.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-20T17:03:03.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-20T17:03:03.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-20T17:03:03.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "epoch": 28, 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "inconsistents": [ 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj1", 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.278 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "version": 18 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "missing" 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj1", 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.279 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1828249343, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'18", 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'1", 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4232.0:1", 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 18, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:37.840221+0000", 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:37.841533+0000", 2026-02-20T17:03:03.280 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.281 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.282 INFO:tasks.workunit.client.0.trial013.stdout: "missing" 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256 2026-02-20T17:03:03.283 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj10", 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "version": 32 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.284 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj10", 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 718195851, 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.285 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'32", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'10", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4253.0:1", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 32, 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.039520+0000", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.039910+0000", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.286 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.287 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.288 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "????", 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.289 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.290 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.291 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj11", 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "version": 34 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.292 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj11", 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 693400951, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'34", 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'11", 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4256.0:1", 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 34, 2026-02-20T17:03:03.293 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.060461+0000", 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.060887+0000", 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.294 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.295 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.296 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.297 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj13", 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "version": 38 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.298 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj13", 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2087409765, 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'38", 2026-02-20T17:03:03.299 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'13", 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4262.0:1", 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 38, 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.115293+0000", 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.115726+0000", 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.300 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.301 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]" 2026-02-20T17:03:03.302 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.303 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.304 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj14", 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "version": 40 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.305 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj14", 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2484217095, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'40", 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'14", 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4265.0:1", 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 40, 2026-02-20T17:03:03.306 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.142486+0000", 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.142849+0000", 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.307 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.308 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1033, 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.309 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.310 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj15", 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.311 INFO:tasks.workunit.client.0.trial013.stdout: "version": 15 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_corrupted" 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj15", 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 612772309, 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.312 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "version": "14'15", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "0'0", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4217.0:1", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 15, 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:37.212275+0000", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:37.212562+0000", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.313 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xee546d8f", 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.314 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_corrupted" 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:03:03.315 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj2", 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "version": 56 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_missing" 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.316 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj2", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1058988552, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "version": "21'56", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "18'48", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4307.0:1", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 56, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0, 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.797419+0000", 2026-02-20T17:03:03.317 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.797730+0000", 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "whiteout", 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "dirty" 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0xffffffff", 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.318 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_missing" 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "size": 0, 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.319 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 7, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 4, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: 4, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: 3, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: 2, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 7, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1024, 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.320 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: 7, 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: 6, 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: 5 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj3", 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: "version": 44 2026-02-20T17:03:03.321 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch" 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [ 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch_info", 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "obj_size_info_mismatch" 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj3", 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1643547569, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "version": "17'44", 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'22", 2026-02-20T17:03:03.322 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4277.0:1", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 44, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.350668+0000", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.351699+0000", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x8ccb9ee7", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.323 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "size_mismatch_info", 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "obj_size_info_mismatch" 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "size": 3840, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "object_info": { 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj3", 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1643547569, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.324 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "version": "17'44", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "15'22", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4277.0:1", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 44, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.350668+0000", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.351699+0000", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x8ccb9ee7", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.325 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "size": 768 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj6", 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.326 INFO:tasks.workunit.client.0.trial013.stdout: "version": 24 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj6", 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2202164420, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'24", 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'6", 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4241.0:1", 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 24, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.327 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:37.924284+0000", 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:37.924644+0000", 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.328 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.329 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj7", 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "version": 26 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj7", 2026-02-20T17:03:03.330 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 1552453721, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'26", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'7", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4244.0:1", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 26, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:37.954578+0000", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:37.954932+0000", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.331 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [] 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.332 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj8", 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.333 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "version": 28 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj8", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 2381834917, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'28", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'8", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4247.0:1", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 28, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:37.983392+0000", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:37.983805+0000", 2026-02-20T17:03:03.334 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.335 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 0, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.336 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "object": { 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "name": "obj9", 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "nspace": "", 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "locator": "", 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "snap": "head", 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: "version": 30 2026-02-20T17:03:03.337 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [ 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "snapset_inconsistency" 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "union_shard_errors": [], 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "selected_object_info": { 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "oid": { 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "oid": "obj9", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "key": "", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "snapid": -2, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "hash": 3833113727, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "max": 0, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "pool": 1, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "namespace": "" 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "version": "15'30", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "prior_version": "14'9", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "last_reqid": "client.4250.0:1", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "user_version": 30, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "mtime": "2026-02-20T17:02:38.011166+0000", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "local_mtime": "2026-02-20T17:02:38.011471+0000", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "lost": 0, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "flags": [ 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "dirty", 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest" 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_seq": 0, 2026-02-20T17:03:03.338 INFO:tasks.workunit.client.0.trial013.stdout: "truncate_size": 0, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "data_digest": "0x65093aa0", 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "omap_digest": "0xffffffff", 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "expected_object_size": 0, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "expected_write_size": 0, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "alloc_hint_flags": 0, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "manifest": { 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "type": 0 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "watchers": {}, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "shard_versions": [] 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "shards": [ 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 0, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "primary": false, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "size": "????", 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.339 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "osd": 1, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "primary": true, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "errors": [], 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "size": 256, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "snapset": { 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "seq": 1, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "clones": [ 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "snap": 1, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "size": 1032, 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "overlap": "[]", 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: "snaps": [ 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: 1 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:03.340 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='513871 ' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='513329 ' 2026-02-20T17:03:03.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-20T17:03:03.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2026-02-20T17:03:03.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-20T17:03:03.492 INFO:tasks.workunit.client.0.trial013.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T17:03:03.492 INFO:tasks.workunit.client.0.trial013.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2026-02-20T17:03:00.300214+0000 21'56 30:107 [1,0] 1 [1,0] 1 21'56 2026-02-20T17:03:00.300156+0000 0'0 2026-02-20T17:02:31.890359+0000 0 1 periodic scrub scheduled @ 2026-02-22T00:44:41.053117+0000 36 0 2026-02-20T17:03:03.492 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:03.492 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T17:03:03.493 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 513871 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 513329 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:03.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:08.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:08.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-20T17:03:08.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-20T17:03:08.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-20T17:03:08.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-20T17:03:08.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2026-02-20T17:03:08.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-20T17:03:08.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2026-02-20T17:03:08.839 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2026-02-20T17:03:08.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2026-02-20T17:03:08.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-20T17:03:08.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-20T17:03:08.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-20T17:03:08.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-20T17:03:08.843 INFO:tasks.workunit.client.0.trial013.stdout:TEST PASSED 2026-02-20T17:03:08.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-20T17:03:08.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=da2f8460-6780-4694-9c46-d990f6ea83b8 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:08.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:08.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:08.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:08.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:03:08.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:03:08.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:03:08.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:03:08.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:03:08.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:03:08.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:08.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:03:08.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:03:08.855 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:08.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:03:08.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:03:08.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:03:08.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:03:08.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:08.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:03:08.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:03:08.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:03:08.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:03:08.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T17:03:08.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-snaps 0 2026-02-20T17:03:08.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:08.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:08.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:08.872 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:03:08.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:03:08.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:03:08.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:03:08.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:03:08.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:03:08.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:08.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:03:08.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:03:08.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:08.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:03:08.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T17:03:08.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-20T17:03:08.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:03:08.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:08.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.459686 2026-02-20T17:03:08.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.459686 2026-02-20T17:03:08.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:03:08.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:03:08.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T17:03:08.883 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T17:03:08.883 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T17:03:08.936 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2026-02-20T17:03:08.936 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2026-02-20T17:03:08.989 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T17:03:08.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-test 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2026-02-20T17:03:08.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2026-02-20T17:03:08.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none ' 2026-02-20T17:03:08.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2026-02-20T17:03:08.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T17:03:08.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2026-02-20T17:03:08.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_abort_periodic_for_operator 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_deep_scrub_abort 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_dump_scrub_schedule 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_interval_changes 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_just_deep_scrubs 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_pg_dump_objects_scrubbed 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_abort 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_permit_time 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:TEST_scrub_test' 2026-02-20T17:03:08.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:03:08.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_abort_periodic_for_operator -------------------' 2026-02-20T17:03:08.999 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_abort_periodic_for_operator ------------------- 2026-02-20T17:03:09.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:03:09.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:09.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:09.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:03:09.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:03:09.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:03:09.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:03:09.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:03:09.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:03:09.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:09.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:03:09.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:03:09.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:09.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:03:09.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:03:09.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:03:09.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:03:09.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:09.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:09.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:03:09.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:03:09.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:03:09.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:03:09.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:03:09.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:09.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:09.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:03:09.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:03:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T17:03:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T17:03:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:03:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:03:09.014 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_abort_periodic_for_operator ----------------------- 2026-02-20T17:03:09.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_abort_periodic_for_operator -----------------------' 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_abort_periodic_for_operator td/osd-scrub-test 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:748: TEST_abort_periodic_for_operator: local dir=td/osd-scrub-test 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: cluster_conf=(['osds_num']='5' ['pgs_in_pool']='16' ['pool_name']='test') 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: local -A cluster_conf 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:754: TEST_abort_periodic_for_operator: local extr_dbg=1 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:756: TEST_abort_periodic_for_operator: standard_scrub_wpq_cluster td/osd-scrub-test cluster_conf 2 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:302: standard_scrub_wpq_cluster: local dir=td/osd-scrub-test 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:303: standard_scrub_wpq_cluster: local -n conf=cluster_conf 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:304: standard_scrub_wpq_cluster: local osd_sleep=2 2026-02-20T17:03:09.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: standard_scrub_wpq_cluster: conf['extras']=' --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: standard_scrub_wpq_cluster: standard_scrub_cluster td/osd-scrub-test conf 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=conf 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=5 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=16 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-20T17:03:09.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-20T17:03:09.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T17:03:09.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T17:03:09.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T17:03:09.294 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 1e79bd29-102c-4169-a3e2-d931499e2499 2026-02-20T17:03:09.395 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:09.424 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:09.423+0000 7f523f43ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:09.425 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:09.423+0000 7f523f43ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:09.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:09.423+0000 7f523f43ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:09.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:09.423+0000 7f523f43ba40 -1 bdev(0x56109e748c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:09.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:09.423+0000 7f523f43ba40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:03:10.241 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:03:10.344 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:03:10.374 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:10.371+0000 7f3b29efaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:10.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:10.375+0000 7f3b29efaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:10.377 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:10.375+0000 7f3b29efaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:10.435 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:10.773 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:10.771+0000 7f3b29efaa40 -1 Falling back to public interface 2026-02-20T17:03:11.524 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:11.750 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:11.747+0000 7f3b29efaa40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:03:12.616 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:03:12.771 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1203602783,v1:127.0.0.1:6801/1203602783] [v2:127.0.0.1:6802/1203602783,v1:127.0.0.1:6803/1203602783] exists,up 1e79bd29-102c-4169-a3e2-d931499e2499 2026-02-20T17:03:12.776 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 ed8709c5-a547-4633-a9da-c85ce941479b 2026-02-20T17:03:12.943 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:12.973 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:12.971+0000 7f33e4e7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:12.974 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:12.971+0000 7f33e4e7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:12.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:12.971+0000 7f33e4e7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:12.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:12.975+0000 7f33e4e7aa40 -1 bdev(0x55a395728c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:12.976 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:12.975+0000 7f33e4e7aa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:03:13.587 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:03:13.784 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:03:13.814 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:13.811+0000 7f6b752c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:13.815 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:13.815+0000 7f6b752c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:13.816 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:13.815+0000 7f6b752c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:13.946 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:14.269 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:14.267+0000 7f6b752c9a40 -1 Falling back to public interface 2026-02-20T17:03:14.877 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:14.875+0000 7f6b752c9a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:03:15.109 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:15.264 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2298876684,v1:127.0.0.1:6811/2298876684] [v2:127.0.0.1:6812/2298876684,v1:127.0.0.1:6813/2298876684] exists,up ed8709c5-a547-4633-a9da-c85ce941479b 2026-02-20T17:03:15.270 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 9cb7bbf5-44c0-40c6-bfae-97b15f79a1ae 2026-02-20T17:03:15.444 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:03:15.484 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:15.479+0000 7fca2607ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:15.485 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:15.483+0000 7fca2607ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:15.486 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:15.483+0000 7fca2607ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:15.486 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:15.483+0000 7fca2607ba40 -1 bdev(0x558acf108c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:15.487 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:15.483+0000 7fca2607ba40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:03:16.143 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:03:16.343 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:03:16.374 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:16.371+0000 7f5b92947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:16.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:16.375+0000 7f5b92947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:16.377 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:16.375+0000 7f5b92947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:16.509 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:16.765 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:16.763+0000 7f5b92947a40 -1 Falling back to public interface 2026-02-20T17:03:17.328 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:17.323+0000 7f5b92947a40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:03:17.663 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:17.818 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3722621780,v1:127.0.0.1:6819/3722621780] [v2:127.0.0.1:6820/3722621780,v1:127.0.0.1:6821/3722621780] exists,up 9cb7bbf5-44c0-40c6-bfae-97b15f79a1ae 2026-02-20T17:03:17.823 INFO:tasks.workunit.client.0.trial013.stdout:add osd3 2bf21dad-21cf-4e4e-8103-8c3c73374dbe 2026-02-20T17:03:18.003 INFO:tasks.workunit.client.0.trial013.stdout:3 2026-02-20T17:03:18.033 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.031+0000 7f061c0f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:18.034 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.031+0000 7f061c0f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:18.035 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.031+0000 7f061c0f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:18.035 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.031+0000 7f061c0f6a40 -1 bdev(0x564f5861cc00 td/osd-scrub-test/3/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:18.036 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.035+0000 7f061c0f6a40 -1 bluestore(td/osd-scrub-test/3) _read_fsid unparsable uuid 2026-02-20T17:03:18.687 INFO:tasks.workunit.client.0.trial013.stdout:adding osd3 key to auth repository 2026-02-20T17:03:18.892 INFO:tasks.workunit.client.0.trial013.stdout:start osd.3 2026-02-20T17:03:18.912 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.907+0000 7fa66cbeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:18.913 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.911+0000 7fa66cbeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:18.914 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:18.911+0000 7fa66cbeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:19.047 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:19.273 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:19.271+0000 7fa66cbeea40 -1 Falling back to public interface 2026-02-20T17:03:19.850 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:19.847+0000 7fa66cbeea40 -1 osd.3 0 log_to_monitors true 2026-02-20T17:03:20.211 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:20.370 INFO:tasks.workunit.client.0.trial013.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1035870567,v1:127.0.0.1:6827/1035870567] [v2:127.0.0.1:6828/1035870567,v1:127.0.0.1:6829/1035870567] exists,up 2bf21dad-21cf-4e4e-8103-8c3c73374dbe 2026-02-20T17:03:20.377 INFO:tasks.workunit.client.0.trial013.stdout:add osd4 25594095-0c2b-435f-a240-e62502845bc3 2026-02-20T17:03:20.551 INFO:tasks.workunit.client.0.trial013.stdout:4 2026-02-20T17:03:20.580 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:20.579+0000 7f1df2f31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:20.582 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:20.579+0000 7f1df2f31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:20.583 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:20.579+0000 7f1df2f31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:20.583 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:20.579+0000 7f1df2f31a40 -1 bdev(0x55c687f46c00 td/osd-scrub-test/4/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:20.583 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:20.583+0000 7f1df2f31a40 -1 bluestore(td/osd-scrub-test/4) _read_fsid unparsable uuid 2026-02-20T17:03:21.240 INFO:tasks.workunit.client.0.trial013.stdout:adding osd4 key to auth repository 2026-02-20T17:03:21.439 INFO:tasks.workunit.client.0.trial013.stdout:start osd.4 2026-02-20T17:03:21.458 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:21.455+0000 7f58df75ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:21.459 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:21.455+0000 7f58df75ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:21.460 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:21.459+0000 7f58df75ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:21.599 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:21.849 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:21.847+0000 7f58df75ca40 -1 Falling back to public interface 2026-02-20T17:03:22.498 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:22.495+0000 7f58df75ca40 -1 osd.4 0 log_to_monitors true 2026-02-20T17:03:22.757 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:22.913 INFO:tasks.workunit.client.0.trial013.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4243128964,v1:127.0.0.1:6835/4243128964] [v2:127.0.0.1:6836/4243128964,v1:127.0.0.1:6837/4243128964] exists,up 25594095-0c2b-435f-a240-e62502845bc3 2026-02-20T17:03:23.120 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:03:24.762 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836493 2026-02-20T17:03:26.099 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672970 2026-02-20T17:03:26.259 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509448 2026-02-20T17:03:26.433 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.3 seq 85899345925 2026-02-20T17:03:26.596 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.4 seq 107374182403 2026-02-20T17:03:28.464 INFO:tasks.workunit.client.0.trial013.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-20T17:03:28.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:757: TEST_abort_periodic_for_operator: local poolid=1 2026-02-20T17:03:28.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:758: TEST_abort_periodic_for_operator: local poolname=test 2026-02-20T17:03:28.465 INFO:tasks.workunit.client.0.trial013.stdout:Pool: test : 1 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:759: TEST_abort_periodic_for_operator: echo 'Pool: test : 1' 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:762: TEST_abort_periodic_for_operator: local saved_echo_flag=x 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:763: TEST_abort_periodic_for_operator: set +x 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:03:28.466 INFO:tasks.workunit.client.0.trial013.stderr:320 bytes copied, 5.5614e-05 s, 5.8 MB/s 2026-02-20T17:03:34.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:776: TEST_abort_periodic_for_operator: declare -A pg_pr 2026-02-20T17:03:34.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:777: TEST_abort_periodic_for_operator: declare -A pg_ac 2026-02-20T17:03:34.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:778: TEST_abort_periodic_for_operator: declare -A pg_po 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:779: TEST_abort_periodic_for_operator: build_pg_dicts td/osd-scrub-test pg_pr pg_ac pg_po - 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:318: build_pg_dicts: local dir=td/osd-scrub-test 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:319: build_pg_dicts: local -n pg_primary_dict=pg_pr 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:320: build_pg_dicts: local -n pg_acting_dict=pg_ac 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:321: build_pg_dicts: local -n pg_pool_dict=pg_po 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:322: build_pg_dicts: local infile=- 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:324: build_pg_dicts: local extr_dbg=0 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:327: build_pg_dicts: local saved_echo_flag=x 2026-02-20T17:03:34.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:328: build_pg_dicts: set +x 2026-02-20T17:03:34.767 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs_brief 2026-02-20T17:03:35.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:780: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:785: TEST_abort_periodic_for_operator: wait_initial_scrubs pg_pr 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:691: wait_initial_scrubs: local -n pg_to_prim_dict=pg_pr 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:692: wait_initial_scrubs: local extr_dbg=1 2026-02-20T17:03:35.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:696: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_min_interval 7200 2026-02-20T17:03:35.519 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:35.520 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:03:35.520 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.527 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:35.528 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:03:35.528 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.534 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:35.534 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:03:35.534 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.542 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:35.542 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:03:35.542 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.550 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:35.550 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:03:35.550 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:697: wait_initial_scrubs: ceph tell 'osd.*' config set osd_deep_scrub_interval 14400 2026-02-20T17:03:35.634 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:35.635 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-20T17:03:35.635 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.642 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:35.642 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-20T17:03:35.642 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.650 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:35.650 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-20T17:03:35.650 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.657 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:35.657 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-20T17:03:35.657 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.665 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:35.665 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-20T17:03:35.665 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:698: wait_initial_scrubs: ceph tell 'osd.*' config set osd_max_scrubs 32 2026-02-20T17:03:35.746 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:35.747 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:35.747 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.754 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:35.754 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:35.754 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.761 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:35.762 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:35.762 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.769 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:35.769 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:35.769 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.776 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:35.776 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:35.777 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:699: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_sleep 1 2026-02-20T17:03:35.858 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:35.859 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:35.859 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.866 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:35.866 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:35.866 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.873 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:35.873 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:35.873 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.880 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:35.880 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:35.880 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.887 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:35.887 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:35.887 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:700: wait_initial_scrubs: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 10 2026-02-20T17:03:35.963 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:35.963 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:35.963 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.970 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:35.970 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:35.970 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.978 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:35.978 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:35.978 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.985 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:35.985 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:35.985 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:35.992 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:35.992 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:35.992 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:701: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_chunk_max 10 2026-02-20T17:03:36.075 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:36.075 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:36.075 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.083 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:36.083 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:36.083 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.090 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:36.090 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:36.090 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.097 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:36.097 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:36.097 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.104 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:36.104 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:36.104 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.8' 2026-02-20T17:03:36.113 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.8 2026-02-20T17:03:36.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.8 scrub 2026-02-20T17:03:36.184 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.185 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.185 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.185 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.185 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.192 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.9 2026-02-20T17:03:36.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.9' 2026-02-20T17:03:36.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.9 scrub 2026-02-20T17:03:36.256 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.256 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.256 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.256 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.256 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.266 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.4 2026-02-20T17:03:36.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.4' 2026-02-20T17:03:36.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.4 scrub 2026-02-20T17:03:36.332 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.332 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.332 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.332 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.333 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.5' 2026-02-20T17:03:36.342 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.5 2026-02-20T17:03:36.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.5 scrub 2026-02-20T17:03:36.407 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.408 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.408 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.408 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.408 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.416 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.6 2026-02-20T17:03:36.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.6' 2026-02-20T17:03:36.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.6 scrub 2026-02-20T17:03:36.492 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.492 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.492 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.493 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.493 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.7' 2026-02-20T17:03:36.500 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.7 2026-02-20T17:03:36.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.7 scrub 2026-02-20T17:03:36.576 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.576 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.576 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.576 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.576 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.0' 2026-02-20T17:03:36.584 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.0 2026-02-20T17:03:36.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.0 scrub 2026-02-20T17:03:36.659 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.659 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.659 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.659 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.660 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.1' 2026-02-20T17:03:36.667 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.1 2026-02-20T17:03:36.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.1 scrub 2026-02-20T17:03:36.740 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.740 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.740 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.740 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.740 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.2' 2026-02-20T17:03:36.748 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.2 2026-02-20T17:03:36.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.2 scrub 2026-02-20T17:03:36.821 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.821 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.821 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.821 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.821 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.3' 2026-02-20T17:03:36.831 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.3 2026-02-20T17:03:36.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.3 scrub 2026-02-20T17:03:36.895 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.895 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.895 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.895 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.896 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.903 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.d 2026-02-20T17:03:36.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.d' 2026-02-20T17:03:36.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.d scrub 2026-02-20T17:03:36.979 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:36.979 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:36.979 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:36.979 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:36.979 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:36.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:36.987 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.e 2026-02-20T17:03:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.e' 2026-02-20T17:03:36.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.e scrub 2026-02-20T17:03:37.060 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:37.060 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:37.060 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:37.060 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:37.060 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:37.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:37.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:37.068 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.f 2026-02-20T17:03:37.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.f' 2026-02-20T17:03:37.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.f scrub 2026-02-20T17:03:37.143 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:37.143 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:37.143 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:37.143 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:37.144 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:37.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:37.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:37.151 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.a 2026-02-20T17:03:37.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.a' 2026-02-20T17:03:37.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.a scrub 2026-02-20T17:03:37.224 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:37.224 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:37.224 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:37.224 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:37.224 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:37.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:37.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:37.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.b' 2026-02-20T17:03:37.232 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.b 2026-02-20T17:03:37.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.b scrub 2026-02-20T17:03:37.304 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:37.304 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:37.304 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:37.305 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:37.305 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-20T17:03:37.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:37.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.c' 2026-02-20T17:03:37.315 INFO:tasks.workunit.client.0.trial013.stdout:Scheduling initial scrub for 1.c 2026-02-20T17:03:37.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.c scrub 2026-02-20T17:03:37.379 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:37.379 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:37.380 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:37.380 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:37.380 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:37.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:708: wait_initial_scrubs: sleep 1 2026-02-20T17:03:38.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:38.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:38.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:710: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:38.536 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.f", 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.e", 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.547 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.d", 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.0", 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.1", 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.548 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.c", 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.2", 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.3", 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.549 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.4", 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.5", 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.6", 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.7", 2026-02-20T17:03:38.550 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.8", 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.9", 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.a", 2026-02-20T17:03:38.551 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:38.552 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:38.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:712: wait_initial_scrubs: tout=20 2026-02-20T17:03:38.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 20 -gt 0 ']' 2026-02-20T17:03:38.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:39.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:39.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:39.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:39.696 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.f", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.e", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.0", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.c", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.3", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.7", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.a", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:39.706 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:39.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:39.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:39.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:39.857 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:39.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=34 2026-02-20T17:03:39.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 34 -le 1 ']' 2026-02-20T17:03:39.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=8 2026-02-20T17:03:39.867 INFO:tasks.workunit.client.0.trial013.stdout:Still waiting for 8 PGs to finish initial scrubs (timeout 20) 2026-02-20T17:03:39.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 8 PGs to finish initial scrubs (timeout 20)' 2026-02-20T17:03:39.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=19 2026-02-20T17:03:39.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 19 -gt 0 ']' 2026-02-20T17:03:39.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:40.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:40.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:40.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.f", 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.e", 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.0", 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.c", 2026-02-20T17:03:41.190 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.3", 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.7", 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.a", 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=34 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 34 -le 1 ']' 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=8 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 8 PGs to finish initial scrubs (timeout 19)' 2026-02-20T17:03:41.191 INFO:tasks.workunit.client.0.trial013.stdout:Still waiting for 8 PGs to finish initial scrubs (timeout 19) 2026-02-20T17:03:41.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=18 2026-02-20T17:03:41.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 18 -gt 0 ']' 2026-02-20T17:03:41.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:42.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:42.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:42.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:42.335 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:42.344 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.e", 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.a", 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:42.345 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:42.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:42.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:42.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:42.491 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:42.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=14 2026-02-20T17:03:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 14 -le 1 ']' 2026-02-20T17:03:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=3 2026-02-20T17:03:42.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 3 PGs to finish initial scrubs (timeout 18)' 2026-02-20T17:03:42.503 INFO:tasks.workunit.client.0.trial013.stdout:Still waiting for 3 PGs to finish initial scrubs (timeout 18) 2026-02-20T17:03:42.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=17 2026-02-20T17:03:42.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 17 -gt 0 ']' 2026-02-20T17:03:42.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:43.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:43.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:43.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:43.648 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:43.658 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:43.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:43.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:43.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:43.803 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-20T17:03:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-20T17:03:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-20T17:03:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 17)' 2026-02-20T17:03:43.813 INFO:tasks.workunit.client.0.trial013.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 17) 2026-02-20T17:03:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=16 2026-02-20T17:03:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 16 -gt 0 ']' 2026-02-20T17:03:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:44.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:44.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:44.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:44.961 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout:[ 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout: "pgid": "1.b", 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout: "last_scrub_duration": 0 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:44.972 INFO:tasks.workunit.client.0.trial013.stdout:] 2026-02-20T17:03:44.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:44.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:44.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:45.119 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:45.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-20T17:03:45.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-20T17:03:45.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-20T17:03:45.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 16)' 2026-02-20T17:03:45.128 INFO:tasks.workunit.client.0.trial013.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 16) 2026-02-20T17:03:45.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=15 2026-02-20T17:03:45.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 15 -gt 0 ']' 2026-02-20T17:03:45.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-20T17:03:46.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-20T17:03:46.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:46.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:46.277 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:46.286 INFO:tasks.workunit.client.0.trial013.stdout:[] 2026-02-20T17:03:46.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:46.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:46.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-20T17:03:46.433 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:46.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=1 2026-02-20T17:03:46.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 1 -le 1 ']' 2026-02-20T17:03:46.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:721: wait_initial_scrubs: break 2026-02-20T17:03:46.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:727: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:03:46.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:728: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-20T17:03:46.586 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:46.597 INFO:tasks.workunit.client.0.trial013.stdout:[] 2026-02-20T17:03:46.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:729: wait_initial_scrubs: (( tout == 0 )) 2026-02-20T17:03:46.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:730: wait_initial_scrubs: return 0 2026-02-20T17:03:46.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:788: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_max_scrubs 1 2026-02-20T17:03:46.665 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:46.665 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:46.665 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.672 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:46.672 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:46.672 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.679 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:46.679 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:46.679 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.686 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:46.686 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:46.686 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.693 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:46.694 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:46.694 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:789: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-20T17:03:46.767 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:46.767 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T17:03:46.767 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.774 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:46.774 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T17:03:46.774 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.781 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:46.781 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T17:03:46.781 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.788 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:46.788 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T17:03:46.788 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.795 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:46.795 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-20T17:03:46.795 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:792: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_sleep 3 2026-02-20T17:03:46.870 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:46.870 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:46.870 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.877 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:46.877 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:46.877 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.884 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:46.884 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:46.884 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.891 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:46.891 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:46.891 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.898 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:46.898 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:03:46.898 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:793: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 2 2026-02-20T17:03:46.975 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:46.975 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:46.975 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.983 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:46.983 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:46.983 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.990 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:46.990 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:46.991 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:46.998 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:46.998 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:46.998 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.005 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:47.005 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:03:47.005 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:794: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_chunk_max 2 2026-02-20T17:03:47.076 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:47.076 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:47.076 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.083 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:47.083 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:47.083 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.090 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:47.090 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:47.090 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.097 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:47.097 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:47.097 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.104 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:47.104 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-20T17:03:47.105 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:795: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:798: TEST_abort_periodic_for_operator: local pg2= 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:799: TEST_abort_periodic_for_operator: for pg1 in "${!pg_pr[@]}" 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.8 == \1\.\8 ]] 2026-02-20T17:03:47.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:802: TEST_abort_periodic_for_operator: continue 2026-02-20T17:03:47.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-20T17:03:47.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.9 == \1\.\8 ]] 2026-02-20T17:03:47.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:804: TEST_abort_periodic_for_operator: [[ 1 == \1 ]] 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: count_common_active 1.9 1.8 pg_ac 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:367: count_common_active: local pg1=1.9 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:368: count_common_active: local pg2=1.8 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:369: count_common_active: local -n pg_acting_dict=pg_ac 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: a1=('1' '4' '2') 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: local -a a1 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: a2=('1' '4' '3') 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: local -a a2 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:374: count_common_active: local -i cnt=0 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-20T17:03:47.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 1 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=1 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 4 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 3 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 1 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 4 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=2 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 3 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 1 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 4 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 3 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:383: count_common_active: printf %d 2 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: local -i common=2 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:806: TEST_abort_periodic_for_operator: [[ 2 -gt 1 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:807: TEST_abort_periodic_for_operator: pg2=1.9 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:808: TEST_abort_periodic_for_operator: break 2 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:814: TEST_abort_periodic_for_operator: [[ -z 1.9 ]] 2026-02-20T17:03:47.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:820: TEST_abort_periodic_for_operator: ceph tell osd.1 config set osd_max_scrubs 2 2026-02-20T17:03:47.178 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:47.178 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_max_scrubs = '' " 2026-02-20T17:03:47.178 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:47.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:821: TEST_abort_periodic_for_operator: echo 'The two PGs to manipulate are 1.8 and 1.9' 2026-02-20T17:03:47.185 INFO:tasks.workunit.client.0.trial013.stdout:The two PGs to manipulate are 1.8 and 1.9 2026-02-20T17:03:47.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:823: TEST_abort_periodic_for_operator: set_query_debug 1.8 2026-02-20T17:03:47.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.8 2026-02-20T17:03:47.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-20T17:03:47.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.8' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-20T17:03:47.336 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs_brief 2026-02-20T17:03:47.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-20T17:03:47.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.8 is 1' 2026-02-20T17:03:47.346 INFO:tasks.workunit.client.0.trial013.stdout:Setting scrub debug data. Primary for 1.8 is 1 2026-02-20T17:03:47.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-20T17:03:47.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-20T17:03:47.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.515468/ceph-osd.1.asok scrubdebug 1.8 set sessions 2026-02-20T17:03:47.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:826: TEST_abort_periodic_for_operator: local is_act 2026-02-20T17:03:47.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: seq 1 3 2026-02-20T17:03:47.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: for i in $( seq 1 3 ) 2026-02-20T17:03:47.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-20T17:03:47.412 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: jq .scrubber.active 2026-02-20T17:03:47.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: is_act=false 2026-02-20T17:03:47.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:830: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-20T17:03:47.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:831: TEST_abort_periodic_for_operator: break 2026-02-20T17:03:47.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:836: TEST_abort_periodic_for_operator: ceph pg dump pgs 2026-02-20T17:03:47.629 INFO:tasks.workunit.client.0.trial013.stdout:{"success":true}PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-20T17:03:47.629 INFO:tasks.workunit.client.0.trial013.stdout:1.f 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-20T17:03:39.843140+0000 29'19 29:45 [3,0,4] 3 [3,0,4] 3 29'19 2026-02-20T17:03:39.843099+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:39.843099+0000 19 0 2026-02-20T17:03:47.629 INFO:tasks.workunit.client.0.trial013.stdout:1.e 20 0 0 0 0 6400 0 0 20 0 20 active+clean 2026-02-20T17:03:41.807827+0000 29'20 29:46 [1,4,2] 1 [1,4,2] 1 29'20 2026-02-20T17:03:41.807763+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:41.807763+0000 20 0 2026-02-20T17:03:47.629 INFO:tasks.workunit.client.0.trial013.stdout:1.d 14 0 0 0 0 4480 0 0 14 0 14 active+clean 2026-02-20T17:03:38.333103+0000 29'14 29:40 [2,4,1] 2 [2,4,1] 2 29'14 2026-02-20T17:03:38.333039+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:38.333039+0000 14 0 2026-02-20T17:03:47.629 INFO:tasks.workunit.client.0.trial013.stdout:1.0 22 0 0 0 0 7040 0 0 22 0 22 active+clean 2026-02-20T17:03:40.829271+0000 29'22 29:53 [1,0,2] 1 [1,0,2] 1 29'22 2026-02-20T17:03:40.829206+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:40.829206+0000 22 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.1 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-20T17:03:38.364427+0000 29'12 29:43 [4,3,0] 4 [4,3,0] 4 29'12 2026-02-20T17:03:38.364373+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:38.364373+0000 12 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.c 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-20T17:03:40.859484+0000 29'15 29:41 [3,0,4] 3 [3,0,4] 3 29'15 2026-02-20T17:03:40.859410+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:40.859410+0000 15 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.2 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-20T17:03:37.873577+0000 29'15 29:45 [0,4,1] 0 [0,4,1] 0 29'15 2026-02-20T17:03:37.873502+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:37.873502+0000 15 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.3 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-20T17:03:39.382577+0000 29'19 29:49 [4,1,2] 4 [4,1,2] 4 29'19 2026-02-20T17:03:39.382532+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:39.382532+0000 19 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.4 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-20T17:03:37.921794+0000 29'15 29:43 [3,0,1] 3 [3,0,1] 3 29'15 2026-02-20T17:03:37.921737+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:37.921737+0000 15 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.5 26 0 0 0 0 8320 0 0 26 0 26 active+clean 2026-02-20T17:03:37.410833+0000 29'26 29:55 [4,2,3] 4 [4,2,3] 4 29'26 2026-02-20T17:03:37.410768+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:37.410768+0000 26 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.6 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-20T17:03:38.875412+0000 29'12 29:40 [3,4,2] 3 [3,4,2] 3 29'12 2026-02-20T17:03:38.875364+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:38.875364+0000 12 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.7 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-20T17:03:39.822132+0000 29'11 29:39 [1,3,0] 1 [1,3,0] 1 29'11 2026-02-20T17:03:39.822059+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:39.822059+0000 11 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.8 28 0 0 0 0 8960 0 0 28 0 28 active+clean 2026-02-20T17:03:37.821547+0000 29'28 29:56 [1,4,3] 1 [1,4,3] 1 29'28 2026-02-20T17:03:37.821510+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:37.821510+0000 28 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.9 8 0 0 0 0 2560 0 0 8 0 8 active+clean 2026-02-20T17:03:38.823225+0000 29'8 29:35 [1,4,2] 1 [1,4,2] 1 29'8 2026-02-20T17:03:38.823153+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:38.823153+0000 8 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.a 9 0 0 0 0 2880 0 0 9 0 9 active+clean 2026-02-20T17:03:42.797487+0000 29'9 29:36 [1,2,4] 1 [1,2,4] 1 29'9 2026-02-20T17:03:42.797426+0000 0'0 2026-02-20T17:03:23.119539+0000 0 1 periodic scrub scheduled @ 2026-02-20T19:03:42.797426+0000 9 0 2026-02-20T17:03:47.630 INFO:tasks.workunit.client.0.trial013.stdout:1.b 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-20T17:03:43.844276+0000 29'11 29:38 [1,0,2] 1 [1,0,2] 1 29'11 2026-02-20T17:03:43.844199+0000 0'0 2026-02-20T17:03:23.119539+0000 0 2 periodic scrub scheduled @ 2026-02-20T19:03:43.844199+0000 11 0 2026-02-20T17:03:47.631 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:47.631 INFO:tasks.workunit.client.0.trial013.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-20T17:03:47.631 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:03:47.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:837: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-20T17:03:47.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:842: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-20T17:03:48.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:844: TEST_abort_periodic_for_operator: echo 'Initiating a periodic scrub of 1.8' 2026-02-20T17:03:48.140 INFO:tasks.workunit.client.0.trial013.stdout:Initiating a periodic scrub of 1.8 2026-02-20T17:03:48.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:845: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:48.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:846: TEST_abort_periodic_for_operator: ceph tell 1.8 schedule-deep-scrub 2026-02-20T17:03:48.205 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:48.206 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T17:03:48.206 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:03:48.206 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-20T10:39:38.205499+0000" 2026-02-20T17:03:48.206 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:48.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:847: TEST_abort_periodic_for_operator: sleep 1 2026-02-20T17:03:49.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:848: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:49.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: seq 1 14 2026-02-20T17:03:49.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: for i in $( seq 1 14 ) 2026-02-20T17:03:49.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:852: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-20T17:03:49.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-20T17:03:49.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: stt='{ 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "active": true, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "epoch_start": "26", 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "start": "1:10000000::::0", 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "end": "MIN", 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "max_end": "MIN", 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "subset_last_update": "0'\''0", 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "deep": true, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "req_scrub": false, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "auto_repair": false, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "check_repair": false, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "deep_scrub_on_error": false, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "priority": 5, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "shallow_errors": 0, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "deep_errors": 0, 2026-02-20T17:03:49.791 INFO:tasks.workunit.client.0.trial013.stderr: "fixed": 0, 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr: "waiting_on_whom": [], 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr: "schedule": "scrubbing", 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr: "is_reserving_replicas": false, 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr: "test_sequence": 5902340 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr:}' 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: jq .active 2026-02-20T17:03:49.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"26",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 5902340 '}' 2026-02-20T17:03:49.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: is_active=true 2026-02-20T17:03:49.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"26",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 5902340 '}' 2026-02-20T17:03:49.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: is_reserving_replicas=false 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:857: TEST_abort_periodic_for_operator: break 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:866: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: (( extr_dbg >= 1 )) 2026-02-20T17:03:49.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: ceph tell 'osd.*' dump_scrub_reservations --format=json-pretty 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: "scrubs_local": 0, 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: "osd_max_scrubs": 1, 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: "remote_scrub_reservations": { 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: "max_allowed": 1, 2026-02-20T17:03:49.878 INFO:tasks.workunit.client.0.trial013.stdout: "min_priority": 0, 2026-02-20T17:03:49.879 INFO:tasks.workunit.client.0.trial013.stdout: "queues": [], 2026-02-20T17:03:49.879 INFO:tasks.workunit.client.0.trial013.stdout: "in_progress": [] 2026-02-20T17:03:49.879 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.879 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout: "scrubs_local": 1, 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout: "osd_max_scrubs": 2, 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout: "remote_scrub_reservations": { 2026-02-20T17:03:49.885 INFO:tasks.workunit.client.0.trial013.stdout: "max_allowed": 2, 2026-02-20T17:03:49.886 INFO:tasks.workunit.client.0.trial013.stdout: "min_priority": 0, 2026-02-20T17:03:49.886 INFO:tasks.workunit.client.0.trial013.stdout: "queues": [], 2026-02-20T17:03:49.886 INFO:tasks.workunit.client.0.trial013.stdout: "in_progress": [] 2026-02-20T17:03:49.886 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.886 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:49.892 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "scrubs_local": 0, 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "osd_max_scrubs": 1, 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "remote_scrub_reservations": { 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "max_allowed": 1, 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "min_priority": 0, 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "queues": [], 2026-02-20T17:03:49.893 INFO:tasks.workunit.client.0.trial013.stdout: "in_progress": [] 2026-02-20T17:03:49.894 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.894 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout:osd.3: { 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: "scrubs_local": 0, 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: "osd_max_scrubs": 1, 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: "remote_scrub_reservations": { 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: "max_allowed": 1, 2026-02-20T17:03:49.900 INFO:tasks.workunit.client.0.trial013.stdout: "min_priority": 0, 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: "queues": [], 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: "in_progress": [ 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: "item": "1.8", 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: "prio": 0, 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: "can_preempt": false 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.901 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout:osd.4: { 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "scrubs_local": 0, 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "osd_max_scrubs": 1, 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "remote_scrub_reservations": { 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "max_allowed": 1, 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "min_priority": 0, 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "queues": [], 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "in_progress": [ 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "item": "1.8", 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "prio": 0, 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: "can_preempt": false 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.907 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:03:49.908 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:03:49.908 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:49.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:876: TEST_abort_periodic_for_operator: set_query_debug 1.9 2026-02-20T17:03:49.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.9 2026-02-20T17:03:49.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-20T17:03:49.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.9' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-20T17:03:50.066 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs_brief 2026-02-20T17:03:50.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-20T17:03:50.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.9 is 1' 2026-02-20T17:03:50.076 INFO:tasks.workunit.client.0.trial013.stdout:Setting scrub debug data. Primary for 1.9 is 1 2026-02-20T17:03:50.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-20T17:03:50.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.515468/ceph-osd.1.asok scrubdebug 1.9 set sessions 2026-02-20T17:03:50.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:877: TEST_abort_periodic_for_operator: ceph tell 1.9 schedule-deep-scrub 2026-02-20T17:03:50.223 INFO:tasks.workunit.client.0.trial013.stdout:{"success":true}{ 2026-02-20T17:03:50.223 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T17:03:50.223 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:03:50.224 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-20T10:39:40.223343+0000" 2026-02-20T17:03:50.224 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:50.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:878: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-20T17:03:50.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:879: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:50.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:880: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:50.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:881: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:50.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:882: TEST_abort_periodic_for_operator: sleep 1 2026-02-20T17:03:51.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:883: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:51.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:884: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:51.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:885: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-20T17:03:51.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-20T17:03:51.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-20T17:03:51.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: local 'stt2={ 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "active": true, 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "epoch_start": "26", 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "start": "MIN", 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "end": "MIN", 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "max_end": "MIN", 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "subset_last_update": "0'\''0", 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "deep": true, 2026-02-20T17:03:51.807 INFO:tasks.workunit.client.0.trial013.stderr: "req_scrub": false, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "auto_repair": false, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "check_repair": false, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "deep_scrub_on_error": false, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "priority": 5, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "shallow_errors": 0, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "deep_errors": 0, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "fixed": 0, 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "waiting_on_whom": [], 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "schedule": "scrubbing", 2026-02-20T17:03:51.808 INFO:tasks.workunit.client.0.trial013.stderr: "is_reserving_replicas": true, 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr: "osd_to_respond": 4, 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr: "duration_seconds": 1, 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr: "requested_in_order": 2, 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr: "num_to_reserve": 2, 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr: "test_sequence": 4153347 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr:}' 2026-02-20T17:03:51.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:889: TEST_abort_periodic_for_operator: local pg2_is_reserving 2026-02-20T17:03:51.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-20T17:03:51.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"26",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' true, '"osd_to_respond":' 4, '"duration_seconds":' 1, '"requested_in_order":' 2, '"num_to_reserve":' 2, '"test_sequence":' 4153347 '}' 2026-02-20T17:03:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: pg2_is_reserving=true 2026-02-20T17:03:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:891: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-20T17:03:51.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:899: TEST_abort_periodic_for_operator: echo 'Instructing 1.9 to perform a high-priority scrub' 2026-02-20T17:03:51.817 INFO:tasks.workunit.client.0.trial013.stdout:Instructing 1.9 to perform a high-priority scrub 2026-02-20T17:03:51.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:900: TEST_abort_periodic_for_operator: ceph tell 1.9 scrub 2026-02-20T17:03:51.880 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:03:51.881 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:03:51.881 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:03:51.881 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:03:51.881 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:03:51.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: seq 1 10 2026-02-20T17:03:51.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: for i in $( seq 1 10 ) 2026-02-20T17:03:51.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:903: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-20T17:03:52.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-20T17:03:52.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-20T17:03:52.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: stt2='{ 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "active": true, 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "epoch_start": "26", 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "start": "1:90000000::::0", 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "end": "MIN", 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "max_end": "MIN", 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "subset_last_update": "0'\''0", 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "deep": false, 2026-02-20T17:03:52.464 INFO:tasks.workunit.client.0.trial013.stderr: "req_scrub": true, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "auto_repair": false, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "check_repair": false, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "deep_scrub_on_error": false, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "priority": 5, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "shallow_errors": 0, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "deep_errors": 0, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "fixed": 0, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "waiting_on_whom": [], 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "schedule": "scrubbing", 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "is_reserving_replicas": false, 2026-02-20T17:03:52.465 INFO:tasks.workunit.client.0.trial013.stderr: "test_sequence": 4153349 2026-02-20T17:03:52.466 INFO:tasks.workunit.client.0.trial013.stderr:}' 2026-02-20T17:03:52.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: jq .active 2026-02-20T17:03:52.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"26",' '"start":' '"1:90000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' false, '"req_scrub":' true, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 4153349 '}' 2026-02-20T17:03:52.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: pg2_is_active=true 2026-02-20T17:03:52.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-20T17:03:52.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"26",' '"start":' '"1:90000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' false, '"req_scrub":' true, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 4153349 '}' 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: pg2_is_reserving=false 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ false != \t\r\u\e ]] 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:908: TEST_abort_periodic_for_operator: break 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ false = \t\r\u\e ]] 2026-02-20T17:03:52.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:917: TEST_abort_periodic_for_operator: echo Done 2026-02-20T17:03:52.486 INFO:tasks.workunit.client.0.trial013.stdout:Done 2026-02-20T17:03:52.486 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_abort_periodic_for_operator ------------------ 2026-02-20T17:03:52.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_abort_periodic_for_operator ------------------' 2026-02-20T17:03:52.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:52.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:52.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:52.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:03:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:03:52.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:03:52.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:03:52.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:03:52.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:03:52.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:52.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:03:52.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:03:52.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:52.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:03:52.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:03:52.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:03:52.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:03:52.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:03:52.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:03:52.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:03:52.830 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_abort_periodic_for_operator ------------------ 2026-02-20T17:03:52.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_abort_periodic_for_operator ------------------' 2026-02-20T17:03:52.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:03:52.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2026-02-20T17:03:52.831 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:03:52.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:03:52.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:03:52.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:03:52.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:03:52.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:03:52.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:03:52.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:03:52.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:03:52.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:52.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:03:52.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:03:52.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:03:52.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:03:52.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:03:52.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:03:52.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:03:52.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:03:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:03:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:03:52.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:03:52.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:03:52.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:03:52.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:409: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-20T17:03:52.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=deep-scrub 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.515468 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test deep-scrub = scrub 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: stopscrub=nodeep-scrub 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: check=nodeep_scrub 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:03:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:52.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:03:52.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:03:52.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:03:52.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:03:52.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:03:52.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:03:52.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:03:52.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:03:52.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:03:52.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:03:52.901 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:03:52.902 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.902 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.902 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:03:52.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:03:52.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:03:52.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:03:52.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:03:52.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:03:52.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:03:52.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:03:52.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:03:53.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:53.103 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:53.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:53.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:03:53.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:03:53.123 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-20T17:03:53.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:03:53.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:53.129 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:03:53.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:03:53.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:03:53.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:03:53.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d7e8938-c8aa-40cc-9e80-12765ee732b3 2026-02-20T17:03:53.132 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 5d7e8938-c8aa-40cc-9e80-12765ee732b3 2026-02-20T17:03:53.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5d7e8938-c8aa-40cc-9e80-12765ee732b3' 2026-02-20T17:03:53.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:03:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5k5hpAU5XCBAAffA5Bmw7NwiDogfW9ZGyzQ== 2026-02-20T17:03:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5k5hpAU5XCBAAffA5Bmw7NwiDogfW9ZGyzQ=="}' 2026-02-20T17:03:53.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d7e8938-c8aa-40cc-9e80-12765ee732b3 -i td/osd-scrub-test/0/new.json 2026-02-20T17:03:53.231 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:03:53.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQD5k5hpAU5XCBAAffA5Bmw7NwiDogfW9ZGyzQ== --osd-uuid 5d7e8938-c8aa-40cc-9e80-12765ee732b3 2026-02-20T17:03:53.257 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:53.254+0000 7feddc568a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:53.259 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:53.254+0000 7feddc568a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:53.260 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:53.258+0000 7feddc568a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:53.261 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:53.258+0000 7feddc568a40 -1 bdev(0x55c952f4ac00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:53.261 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:53.258+0000 7feddc568a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:03:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:03:54.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:03:54.063 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:03:54.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:03:54.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:03:54.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:03:54.168 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:03:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:03:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:03:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:03:54.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:03:54.183 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:54.178+0000 7ff3a2286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:54.184 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:54.182+0000 7ff3a2286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:54.185 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:54.182+0000 7ff3a2286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:54.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:03:54.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:03:54.260 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:03:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:03:54.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:03:54.597 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:54.594+0000 7ff3a2286a40 -1 Falling back to public interface 2026-02-20T17:03:55.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:03:55.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:03:55.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:03:55.349 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:55.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:03:55.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:03:55.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:03:55.541 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:55.538+0000 7ff3a2286a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:03:56.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:03:56.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:03:56.440 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:03:56.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:03:56.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:03:56.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:03:56.596 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/877609360,v1:127.0.0.1:6801/877609360] [v2:127.0.0.1:6802/877609360,v1:127.0.0.1:6803/877609360] exists,up 5d7e8938-c8aa-40cc-9e80-12765ee732b3 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:03:56.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:03:56.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:03:56.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:03:56.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:03:56.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e7378fd-bd5b-47b0-ac3e-0662e412e9e7 2026-02-20T17:03:56.604 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 1e7378fd-bd5b-47b0-ac3e-0662e412e9e7 2026-02-20T17:03:56.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1e7378fd-bd5b-47b0-ac3e-0662e412e9e7' 2026-02-20T17:03:56.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:03:56.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8k5hpFrmPJBAAX3GT8DZnO/hgBSm8HkH2Zg== 2026-02-20T17:03:56.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8k5hpFrmPJBAAX3GT8DZnO/hgBSm8HkH2Zg=="}' 2026-02-20T17:03:56.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e7378fd-bd5b-47b0-ac3e-0662e412e9e7 -i td/osd-scrub-test/1/new.json 2026-02-20T17:03:56.773 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:56.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:03:56.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQD8k5hpFrmPJBAAX3GT8DZnO/hgBSm8HkH2Zg== --osd-uuid 1e7378fd-bd5b-47b0-ac3e-0662e412e9e7 2026-02-20T17:03:56.800 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:56.798+0000 7fd6905aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:56.802 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:56.798+0000 7fd6905aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:56.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:56.798+0000 7fd6905aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:56.803 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:56.798+0000 7fd6905aaa40 -1 bdev(0x55613fad4c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:56.804 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:56.798+0000 7fd6905aaa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:03:57.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:03:57.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:03:57.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:03:57.476 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:03:57.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:03:57.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:03:57.665 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:03:57.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:03:57.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:03:57.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:03:57.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:03:57.689 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:57.686+0000 7f2a8dff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:57.691 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:57.686+0000 7f2a8dff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:57.692 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:57.690+0000 7f2a8dff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:57.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:03:57.826 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:03:57.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:03:57.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:03:57.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:03:57.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:03:58.089 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:58.086+0000 7f2a8dff8a40 -1 Falling back to public interface 2026-02-20T17:03:58.642 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:58.638+0000 7f2a8dff8a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:03:58.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:03:58.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:03:58.987 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:03:58.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:03:58.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:03:58.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:03:59.141 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3687236974,v1:127.0.0.1:6811/3687236974] [v2:127.0.0.1:6812/3687236974,v1:127.0.0.1:6813/3687236974] exists,up 1e7378fd-bd5b-47b0-ac3e-0662e412e9e7 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:03:59.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:03:59.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:03:59.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:03:59.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:03:59.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5362e65c-c204-42b6-8048-d190ea5d3f5c 2026-02-20T17:03:59.146 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 5362e65c-c204-42b6-8048-d190ea5d3f5c 2026-02-20T17:03:59.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5362e65c-c204-42b6-8048-d190ea5d3f5c' 2026-02-20T17:03:59.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:03:59.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/k5hp3HxlCRAADlp9fOHu9q2tHsiJJy4PRQ== 2026-02-20T17:03:59.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/k5hp3HxlCRAADlp9fOHu9q2tHsiJJy4PRQ=="}' 2026-02-20T17:03:59.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5362e65c-c204-42b6-8048-d190ea5d3f5c -i td/osd-scrub-test/2/new.json 2026-02-20T17:03:59.316 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:03:59.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:03:59.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQD/k5hp3HxlCRAADlp9fOHu9q2tHsiJJy4PRQ== --osd-uuid 5362e65c-c204-42b6-8048-d190ea5d3f5c 2026-02-20T17:03:59.344 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:59.342+0000 7fdf375c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:59.345 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:59.342+0000 7fdf375c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:59.346 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:59.342+0000 7fdf375c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:03:59.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:59.342+0000 7fdf375c7a40 -1 bdev(0x561161d06c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:03:59.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:03:59.342+0000 7fdf375c7a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:04:00.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:04:00.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:04:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:04:00.020 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:04:00.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:04:00.218 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:04:00.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:04:00.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:04:00.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:04:00.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:04:00.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:04:00.243 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:04:00.238+0000 7fbe610a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:04:00.245 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:04:00.242+0000 7fbe610a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:04:00.246 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:04:00.242+0000 7fbe610a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:04:00.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:04:00.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:04:00.381 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:04:00.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:04:00.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:04:00.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:04:00.629 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:04:00.626+0000 7fbe610a2a40 -1 Falling back to public interface 2026-02-20T17:04:01.171 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:04:01.166+0000 7fbe610a2a40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:04:01.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:04:01.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:04:01.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:04:01.540 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:04:01.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:04:01.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:04:01.694 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/478570491,v1:127.0.0.1:6819/478570491] [v2:127.0.0.1:6820/478570491,v1:127.0.0.1:6821/478570491] exists,up 5362e65c-c204-42b6-8048-d190ea5d3f5c 2026-02-20T17:04:01.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:04:01.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:04:01.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:04:01.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-20T17:04:01.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:04:01.898 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:04:01.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:04:02.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-20T17:04:02.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:04:02.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:04:02.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:04:02.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:04:02.913 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:04:02.913 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:04:02.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:04:02.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:04:02.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:04:03.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:04:03.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:04:03.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:04:03.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:04:03.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:03.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:03.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:03.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T17:04:03.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T17:04:03.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T17:04:03.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:03.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:03.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-20T17:04:03.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-20T17:04:03.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2026-02-20T17:04:03.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:03.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:03.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-20T17:04:03.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-20T17:04:03.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509443' 2026-02-20T17:04:03.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:03.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T17:04:03.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:03.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:03.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T17:04:03.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:03.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T17:04:03.402 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T17:04:03.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T17:04:03.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:03.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T17:04:03.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:03.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-20T17:04:03.574 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:03.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:03.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-20T17:04:03.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:03.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-20T17:04:03.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-20T17:04:03.578 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672965 2026-02-20T17:04:03.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:03.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-20T17:04:03.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:03.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-20T17:04:03.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:03.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:03.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-20T17:04:03.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:03.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-20T17:04:03.749 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509443 2026-02-20T17:04:03.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-20T17:04:03.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509443 2026-02-20T17:04:03.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:04.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:04.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:05.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509443 2026-02-20T17:04:05.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:06.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:04:06.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:06.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509443 2026-02-20T17:04:06.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:04:06.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:04:06.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:04:06.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:04:06.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:04:06.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:04:06.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:04:06.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:04:06.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:04:06.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:04:06.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:04:06.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:04:06.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-20T17:04:06.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-20T17:04:06.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-20T17:04:06.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-20T17:04:06.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:04:06.957 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:04:06.957 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:04:06.957 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.4805e-05 s, 29.7 MB/s 2026-02-20T17:04:06.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-20T17:04:06.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:06.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.515468 2026-02-20T17:04:06.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:06.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.515468 2026-02-20T17:04:07.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.515468 2026-02-20T17:04:07.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.515468 2026-02-20T17:04:07.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.515468 2026-02-20T17:04:07.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.515468 2026-02-20T17:04:07.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.515468 2026-02-20T17:04:07.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.515468 2026-02-20T17:04:07.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.515468 2026-02-20T17:04:07.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.515468 2026-02-20T17:04:07.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.515468 2026-02-20T17:04:07.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.515468 2026-02-20T17:04:07.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.515468 2026-02-20T17:04:07.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.515468 2026-02-20T17:04:07.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.515468 2026-02-20T17:04:07.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.515468 2026-02-20T17:04:07.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.515468 2026-02-20T17:04:07.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.515468 2026-02-20T17:04:07.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.515468 2026-02-20T17:04:07.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.515468 2026-02-20T17:04:07.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.515468 2026-02-20T17:04:07.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.515468 2026-02-20T17:04:07.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.515468 2026-02-20T17:04:07.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.515468 2026-02-20T17:04:07.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.515468 2026-02-20T17:04:07.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.515468 2026-02-20T17:04:07.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.515468 2026-02-20T17:04:07.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.515468 2026-02-20T17:04:07.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.515468 2026-02-20T17:04:07.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.515468 2026-02-20T17:04:07.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.515468 2026-02-20T17:04:07.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.515468 2026-02-20T17:04:07.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.515468 2026-02-20T17:04:07.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.515468 2026-02-20T17:04:07.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.515468 2026-02-20T17:04:07.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.515468 2026-02-20T17:04:07.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.515468 2026-02-20T17:04:07.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.515468 2026-02-20T17:04:07.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.515468 2026-02-20T17:04:07.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.515468 2026-02-20T17:04:07.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.515468 2026-02-20T17:04:07.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.515468 2026-02-20T17:04:07.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.515468 2026-02-20T17:04:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.515468 2026-02-20T17:04:07.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:07.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.515468 2026-02-20T17:04:08.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.515468 2026-02-20T17:04:08.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.515468 2026-02-20T17:04:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.515468 2026-02-20T17:04:08.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.515468 2026-02-20T17:04:08.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.515468 2026-02-20T17:04:08.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.515468 2026-02-20T17:04:08.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.515468 2026-02-20T17:04:08.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.515468 2026-02-20T17:04:08.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.515468 2026-02-20T17:04:08.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.515468 2026-02-20T17:04:08.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.515468 2026-02-20T17:04:08.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.515468 2026-02-20T17:04:08.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.515468 2026-02-20T17:04:08.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.515468 2026-02-20T17:04:08.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.515468 2026-02-20T17:04:08.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.515468 2026-02-20T17:04:08.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.515468 2026-02-20T17:04:08.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.515468 2026-02-20T17:04:08.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.515468 2026-02-20T17:04:08.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.515468 2026-02-20T17:04:08.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.515468 2026-02-20T17:04:08.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.515468 2026-02-20T17:04:08.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.515468 2026-02-20T17:04:08.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.515468 2026-02-20T17:04:08.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.515468 2026-02-20T17:04:08.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.515468 2026-02-20T17:04:08.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.515468 2026-02-20T17:04:08.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.515468 2026-02-20T17:04:08.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.515468 2026-02-20T17:04:08.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.515468 2026-02-20T17:04:08.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.515468 2026-02-20T17:04:08.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.515468 2026-02-20T17:04:08.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.515468 2026-02-20T17:04:08.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.515468 2026-02-20T17:04:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.515468 2026-02-20T17:04:08.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.515468 2026-02-20T17:04:08.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.515468 2026-02-20T17:04:08.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.515468 2026-02-20T17:04:08.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.515468 2026-02-20T17:04:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.515468 2026-02-20T17:04:08.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.515468 2026-02-20T17:04:08.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:08.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.515468 2026-02-20T17:04:09.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.515468 2026-02-20T17:04:09.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.515468 2026-02-20T17:04:09.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.515468 2026-02-20T17:04:09.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.515468 2026-02-20T17:04:09.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.515468 2026-02-20T17:04:09.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.515468 2026-02-20T17:04:09.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.515468 2026-02-20T17:04:09.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.515468 2026-02-20T17:04:09.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.515468 2026-02-20T17:04:09.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.515468 2026-02-20T17:04:09.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.515468 2026-02-20T17:04:09.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.515468 2026-02-20T17:04:09.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.515468 2026-02-20T17:04:09.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.515468 2026-02-20T17:04:09.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.515468 2026-02-20T17:04:09.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.515468 2026-02-20T17:04:09.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.515468 2026-02-20T17:04:09.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.515468 2026-02-20T17:04:09.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.515468 2026-02-20T17:04:09.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.515468 2026-02-20T17:04:09.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.515468 2026-02-20T17:04:09.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.515468 2026-02-20T17:04:09.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.515468 2026-02-20T17:04:09.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.515468 2026-02-20T17:04:09.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.515468 2026-02-20T17:04:09.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.515468 2026-02-20T17:04:09.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.515468 2026-02-20T17:04:09.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.515468 2026-02-20T17:04:09.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.515468 2026-02-20T17:04:09.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.515468 2026-02-20T17:04:09.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.515468 2026-02-20T17:04:09.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.515468 2026-02-20T17:04:09.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.515468 2026-02-20T17:04:09.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.515468 2026-02-20T17:04:09.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.515468 2026-02-20T17:04:09.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.515468 2026-02-20T17:04:09.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.515468 2026-02-20T17:04:09.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.515468 2026-02-20T17:04:09.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.515468 2026-02-20T17:04:09.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.515468 2026-02-20T17:04:09.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.515468 2026-02-20T17:04:09.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.515468 2026-02-20T17:04:09.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:09.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.515468 2026-02-20T17:04:10.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.515468 2026-02-20T17:04:10.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.515468 2026-02-20T17:04:10.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.515468 2026-02-20T17:04:10.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.515468 2026-02-20T17:04:10.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.515468 2026-02-20T17:04:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.515468 2026-02-20T17:04:10.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.515468 2026-02-20T17:04:10.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.515468 2026-02-20T17:04:10.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.515468 2026-02-20T17:04:10.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.515468 2026-02-20T17:04:10.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.515468 2026-02-20T17:04:10.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.515468 2026-02-20T17:04:10.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.515468 2026-02-20T17:04:10.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.515468 2026-02-20T17:04:10.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.515468 2026-02-20T17:04:10.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.515468 2026-02-20T17:04:10.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.515468 2026-02-20T17:04:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.515468 2026-02-20T17:04:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.515468 2026-02-20T17:04:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.515468 2026-02-20T17:04:10.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.515468 2026-02-20T17:04:10.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.515468 2026-02-20T17:04:10.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.515468 2026-02-20T17:04:10.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.515468 2026-02-20T17:04:10.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.515468 2026-02-20T17:04:10.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.515468 2026-02-20T17:04:10.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.515468 2026-02-20T17:04:10.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.515468 2026-02-20T17:04:10.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.515468 2026-02-20T17:04:10.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.515468 2026-02-20T17:04:10.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.515468 2026-02-20T17:04:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.515468 2026-02-20T17:04:10.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.515468 2026-02-20T17:04:10.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.515468 2026-02-20T17:04:10.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.515468 2026-02-20T17:04:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.515468 2026-02-20T17:04:10.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.515468 2026-02-20T17:04:10.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.515468 2026-02-20T17:04:10.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.515468 2026-02-20T17:04:10.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.515468 2026-02-20T17:04:10.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.515468 2026-02-20T17:04:10.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:10.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.515468 2026-02-20T17:04:11.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.515468 2026-02-20T17:04:11.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.515468 2026-02-20T17:04:11.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.515468 2026-02-20T17:04:11.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.515468 2026-02-20T17:04:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.515468 2026-02-20T17:04:11.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.515468 2026-02-20T17:04:11.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.515468 2026-02-20T17:04:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.515468 2026-02-20T17:04:11.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.515468 2026-02-20T17:04:11.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.515468 2026-02-20T17:04:11.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.515468 2026-02-20T17:04:11.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.515468 2026-02-20T17:04:11.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.515468 2026-02-20T17:04:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.515468 2026-02-20T17:04:11.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.515468 2026-02-20T17:04:11.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.515468 2026-02-20T17:04:11.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.515468 2026-02-20T17:04:11.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.515468 2026-02-20T17:04:11.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.515468 2026-02-20T17:04:11.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.515468 2026-02-20T17:04:11.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.515468 2026-02-20T17:04:11.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.515468 2026-02-20T17:04:11.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.515468 2026-02-20T17:04:11.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.515468 2026-02-20T17:04:11.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.515468 2026-02-20T17:04:11.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.515468 2026-02-20T17:04:11.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.515468 2026-02-20T17:04:11.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.515468 2026-02-20T17:04:11.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.515468 2026-02-20T17:04:11.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.515468 2026-02-20T17:04:11.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.515468 2026-02-20T17:04:11.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.515468 2026-02-20T17:04:11.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.515468 2026-02-20T17:04:11.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.515468 2026-02-20T17:04:11.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.515468 2026-02-20T17:04:11.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.515468 2026-02-20T17:04:11.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.515468 2026-02-20T17:04:11.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.515468 2026-02-20T17:04:11.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.515468 2026-02-20T17:04:11.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.515468 2026-02-20T17:04:11.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.515468 2026-02-20T17:04:11.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:11.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.515468 2026-02-20T17:04:12.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.515468 2026-02-20T17:04:12.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.515468 2026-02-20T17:04:12.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.515468 2026-02-20T17:04:12.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.515468 2026-02-20T17:04:12.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.515468 2026-02-20T17:04:12.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.515468 2026-02-20T17:04:12.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.515468 2026-02-20T17:04:12.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.515468 2026-02-20T17:04:12.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.515468 2026-02-20T17:04:12.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.515468 2026-02-20T17:04:12.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.515468 2026-02-20T17:04:12.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.515468 2026-02-20T17:04:12.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.515468 2026-02-20T17:04:12.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.515468 2026-02-20T17:04:12.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.515468 2026-02-20T17:04:12.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.515468 2026-02-20T17:04:12.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.515468 2026-02-20T17:04:12.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.515468 2026-02-20T17:04:12.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.515468 2026-02-20T17:04:12.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.515468 2026-02-20T17:04:12.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.515468 2026-02-20T17:04:12.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.515468 2026-02-20T17:04:12.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.515468 2026-02-20T17:04:12.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.515468 2026-02-20T17:04:12.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.515468 2026-02-20T17:04:12.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.515468 2026-02-20T17:04:12.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.515468 2026-02-20T17:04:12.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.515468 2026-02-20T17:04:12.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.515468 2026-02-20T17:04:12.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.515468 2026-02-20T17:04:12.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.515468 2026-02-20T17:04:12.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.515468 2026-02-20T17:04:12.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.515468 2026-02-20T17:04:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.515468 2026-02-20T17:04:12.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.515468 2026-02-20T17:04:12.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.515468 2026-02-20T17:04:12.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.515468 2026-02-20T17:04:12.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.515468 2026-02-20T17:04:12.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.515468 2026-02-20T17:04:12.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.515468 2026-02-20T17:04:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.515468 2026-02-20T17:04:12.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.515468 2026-02-20T17:04:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:12.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.515468 2026-02-20T17:04:13.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.515468 2026-02-20T17:04:13.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.515468 2026-02-20T17:04:13.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.515468 2026-02-20T17:04:13.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.515468 2026-02-20T17:04:13.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.515468 2026-02-20T17:04:13.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.515468 2026-02-20T17:04:13.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.515468 2026-02-20T17:04:13.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.515468 2026-02-20T17:04:13.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.515468 2026-02-20T17:04:13.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.515468 2026-02-20T17:04:13.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.515468 2026-02-20T17:04:13.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.515468 2026-02-20T17:04:13.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.515468 2026-02-20T17:04:13.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.515468 2026-02-20T17:04:13.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.515468 2026-02-20T17:04:13.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.515468 2026-02-20T17:04:13.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.515468 2026-02-20T17:04:13.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.515468 2026-02-20T17:04:13.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.515468 2026-02-20T17:04:13.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.515468 2026-02-20T17:04:13.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.515468 2026-02-20T17:04:13.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.515468 2026-02-20T17:04:13.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.515468 2026-02-20T17:04:13.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.515468 2026-02-20T17:04:13.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.515468 2026-02-20T17:04:13.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.515468 2026-02-20T17:04:13.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.515468 2026-02-20T17:04:13.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.515468 2026-02-20T17:04:13.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.515468 2026-02-20T17:04:13.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.515468 2026-02-20T17:04:13.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.515468 2026-02-20T17:04:13.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.515468 2026-02-20T17:04:13.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.515468 2026-02-20T17:04:13.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.515468 2026-02-20T17:04:13.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.515468 2026-02-20T17:04:13.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.515468 2026-02-20T17:04:13.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.515468 2026-02-20T17:04:13.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.515468 2026-02-20T17:04:13.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.515468 2026-02-20T17:04:13.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.515468 2026-02-20T17:04:13.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.515468 2026-02-20T17:04:13.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.515468 2026-02-20T17:04:13.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:13.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.515468 2026-02-20T17:04:14.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.515468 2026-02-20T17:04:14.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.515468 2026-02-20T17:04:14.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.515468 2026-02-20T17:04:14.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.515468 2026-02-20T17:04:14.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.515468 2026-02-20T17:04:14.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.515468 2026-02-20T17:04:14.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.515468 2026-02-20T17:04:14.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.515468 2026-02-20T17:04:14.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.515468 2026-02-20T17:04:14.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.515468 2026-02-20T17:04:14.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.515468 2026-02-20T17:04:14.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.515468 2026-02-20T17:04:14.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.515468 2026-02-20T17:04:14.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.515468 2026-02-20T17:04:14.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.515468 2026-02-20T17:04:14.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.515468 2026-02-20T17:04:14.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.515468 2026-02-20T17:04:14.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.515468 2026-02-20T17:04:14.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.515468 2026-02-20T17:04:14.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.515468 2026-02-20T17:04:14.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.515468 2026-02-20T17:04:14.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.515468 2026-02-20T17:04:14.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.515468 2026-02-20T17:04:14.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.515468 2026-02-20T17:04:14.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.515468 2026-02-20T17:04:14.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.515468 2026-02-20T17:04:14.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.515468 2026-02-20T17:04:14.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.515468 2026-02-20T17:04:14.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.515468 2026-02-20T17:04:14.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.515468 2026-02-20T17:04:14.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.515468 2026-02-20T17:04:14.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.515468 2026-02-20T17:04:14.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.515468 2026-02-20T17:04:14.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.515468 2026-02-20T17:04:14.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.515468 2026-02-20T17:04:14.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.515468 2026-02-20T17:04:14.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.515468 2026-02-20T17:04:14.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.515468 2026-02-20T17:04:14.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.515468 2026-02-20T17:04:14.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.515468 2026-02-20T17:04:14.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.515468 2026-02-20T17:04:14.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.515468 2026-02-20T17:04:14.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:14.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.515468 2026-02-20T17:04:15.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.515468 2026-02-20T17:04:15.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.515468 2026-02-20T17:04:15.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.515468 2026-02-20T17:04:15.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.515468 2026-02-20T17:04:15.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.515468 2026-02-20T17:04:15.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.515468 2026-02-20T17:04:15.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.515468 2026-02-20T17:04:15.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.515468 2026-02-20T17:04:15.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.515468 2026-02-20T17:04:15.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.515468 2026-02-20T17:04:15.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.515468 2026-02-20T17:04:15.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.515468 2026-02-20T17:04:15.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.515468 2026-02-20T17:04:15.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.515468 2026-02-20T17:04:15.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.515468 2026-02-20T17:04:15.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.515468 2026-02-20T17:04:15.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.515468 2026-02-20T17:04:15.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.515468 2026-02-20T17:04:15.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.515468 2026-02-20T17:04:15.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.515468 2026-02-20T17:04:15.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.515468 2026-02-20T17:04:15.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.515468 2026-02-20T17:04:15.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.515468 2026-02-20T17:04:15.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.515468 2026-02-20T17:04:15.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.515468 2026-02-20T17:04:15.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.515468 2026-02-20T17:04:15.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.515468 2026-02-20T17:04:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.515468 2026-02-20T17:04:15.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.515468 2026-02-20T17:04:15.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.515468 2026-02-20T17:04:15.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.515468 2026-02-20T17:04:15.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.515468 2026-02-20T17:04:15.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.515468 2026-02-20T17:04:15.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.515468 2026-02-20T17:04:15.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.515468 2026-02-20T17:04:15.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.515468 2026-02-20T17:04:15.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.515468 2026-02-20T17:04:15.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.515468 2026-02-20T17:04:15.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.515468 2026-02-20T17:04:15.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.515468 2026-02-20T17:04:15.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.515468 2026-02-20T17:04:15.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:15.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.515468 2026-02-20T17:04:16.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.515468 2026-02-20T17:04:16.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.515468 2026-02-20T17:04:16.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.515468 2026-02-20T17:04:16.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.515468 2026-02-20T17:04:16.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.515468 2026-02-20T17:04:16.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.515468 2026-02-20T17:04:16.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.515468 2026-02-20T17:04:16.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.515468 2026-02-20T17:04:16.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.515468 2026-02-20T17:04:16.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.515468 2026-02-20T17:04:16.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.515468 2026-02-20T17:04:16.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.515468 2026-02-20T17:04:16.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.515468 2026-02-20T17:04:16.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.515468 2026-02-20T17:04:16.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.515468 2026-02-20T17:04:16.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.515468 2026-02-20T17:04:16.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.515468 2026-02-20T17:04:16.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.515468 2026-02-20T17:04:16.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.515468 2026-02-20T17:04:16.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.515468 2026-02-20T17:04:16.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.515468 2026-02-20T17:04:16.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.515468 2026-02-20T17:04:16.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.515468 2026-02-20T17:04:16.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.515468 2026-02-20T17:04:16.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.515468 2026-02-20T17:04:16.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.515468 2026-02-20T17:04:16.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.515468 2026-02-20T17:04:16.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.515468 2026-02-20T17:04:16.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.515468 2026-02-20T17:04:16.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.515468 2026-02-20T17:04:16.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.515468 2026-02-20T17:04:16.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.515468 2026-02-20T17:04:16.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.515468 2026-02-20T17:04:16.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.515468 2026-02-20T17:04:16.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.515468 2026-02-20T17:04:16.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.515468 2026-02-20T17:04:16.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.515468 2026-02-20T17:04:16.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.515468 2026-02-20T17:04:16.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.515468 2026-02-20T17:04:16.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.515468 2026-02-20T17:04:16.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.515468 2026-02-20T17:04:16.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.515468 2026-02-20T17:04:16.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.515468 2026-02-20T17:04:16.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:16.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.515468 2026-02-20T17:04:17.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.515468 2026-02-20T17:04:17.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.515468 2026-02-20T17:04:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.515468 2026-02-20T17:04:17.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.515468 2026-02-20T17:04:17.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.515468 2026-02-20T17:04:17.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.515468 2026-02-20T17:04:17.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.515468 2026-02-20T17:04:17.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.515468 2026-02-20T17:04:17.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.515468 2026-02-20T17:04:17.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.515468 2026-02-20T17:04:17.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.515468 2026-02-20T17:04:17.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.515468 2026-02-20T17:04:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.515468 2026-02-20T17:04:17.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.515468 2026-02-20T17:04:17.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.515468 2026-02-20T17:04:17.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.515468 2026-02-20T17:04:17.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.515468 2026-02-20T17:04:17.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.515468 2026-02-20T17:04:17.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.515468 2026-02-20T17:04:17.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.515468 2026-02-20T17:04:17.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.515468 2026-02-20T17:04:17.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.515468 2026-02-20T17:04:17.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.515468 2026-02-20T17:04:17.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.515468 2026-02-20T17:04:17.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.515468 2026-02-20T17:04:17.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.515468 2026-02-20T17:04:17.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.515468 2026-02-20T17:04:17.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.515468 2026-02-20T17:04:17.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.515468 2026-02-20T17:04:17.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.515468 2026-02-20T17:04:17.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.515468 2026-02-20T17:04:17.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.515468 2026-02-20T17:04:17.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.515468 2026-02-20T17:04:17.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.515468 2026-02-20T17:04:17.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.515468 2026-02-20T17:04:17.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.515468 2026-02-20T17:04:17.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.515468 2026-02-20T17:04:17.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.515468 2026-02-20T17:04:17.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.515468 2026-02-20T17:04:17.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.515468 2026-02-20T17:04:17.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.515468 2026-02-20T17:04:17.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:17.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.515468 2026-02-20T17:04:18.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.515468 2026-02-20T17:04:18.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.515468 2026-02-20T17:04:18.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.515468 2026-02-20T17:04:18.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.515468 2026-02-20T17:04:18.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.515468 2026-02-20T17:04:18.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.515468 2026-02-20T17:04:18.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.515468 2026-02-20T17:04:18.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.515468 2026-02-20T17:04:18.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.515468 2026-02-20T17:04:18.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.515468 2026-02-20T17:04:18.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.515468 2026-02-20T17:04:18.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.515468 2026-02-20T17:04:18.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.515468 2026-02-20T17:04:18.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.515468 2026-02-20T17:04:18.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.515468 2026-02-20T17:04:18.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.515468 2026-02-20T17:04:18.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.515468 2026-02-20T17:04:18.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.515468 2026-02-20T17:04:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.515468 2026-02-20T17:04:18.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.515468 2026-02-20T17:04:18.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.515468 2026-02-20T17:04:18.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.515468 2026-02-20T17:04:18.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.515468 2026-02-20T17:04:18.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.515468 2026-02-20T17:04:18.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.515468 2026-02-20T17:04:18.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.515468 2026-02-20T17:04:18.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.515468 2026-02-20T17:04:18.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.515468 2026-02-20T17:04:18.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.515468 2026-02-20T17:04:18.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.515468 2026-02-20T17:04:18.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.515468 2026-02-20T17:04:18.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.515468 2026-02-20T17:04:18.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.515468 2026-02-20T17:04:18.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.515468 2026-02-20T17:04:18.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.515468 2026-02-20T17:04:18.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.515468 2026-02-20T17:04:18.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.515468 2026-02-20T17:04:18.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.515468 2026-02-20T17:04:18.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.515468 2026-02-20T17:04:18.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.515468 2026-02-20T17:04:18.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.515468 2026-02-20T17:04:18.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.515468 2026-02-20T17:04:18.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.515468 2026-02-20T17:04:18.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:18.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.515468 2026-02-20T17:04:19.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.515468 2026-02-20T17:04:19.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.515468 2026-02-20T17:04:19.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.515468 2026-02-20T17:04:19.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.515468 2026-02-20T17:04:19.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.515468 2026-02-20T17:04:19.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.515468 2026-02-20T17:04:19.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.515468 2026-02-20T17:04:19.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.515468 2026-02-20T17:04:19.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.515468 2026-02-20T17:04:19.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.515468 2026-02-20T17:04:19.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.515468 2026-02-20T17:04:19.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.515468 2026-02-20T17:04:19.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.515468 2026-02-20T17:04:19.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.515468 2026-02-20T17:04:19.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.515468 2026-02-20T17:04:19.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.515468 2026-02-20T17:04:19.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.515468 2026-02-20T17:04:19.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.515468 2026-02-20T17:04:19.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.515468 2026-02-20T17:04:19.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.515468 2026-02-20T17:04:19.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.515468 2026-02-20T17:04:19.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.515468 2026-02-20T17:04:19.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.515468 2026-02-20T17:04:19.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.515468 2026-02-20T17:04:19.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.515468 2026-02-20T17:04:19.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.515468 2026-02-20T17:04:19.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.515468 2026-02-20T17:04:19.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.515468 2026-02-20T17:04:19.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.515468 2026-02-20T17:04:19.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.515468 2026-02-20T17:04:19.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.515468 2026-02-20T17:04:19.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.515468 2026-02-20T17:04:19.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.515468 2026-02-20T17:04:19.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.515468 2026-02-20T17:04:19.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.515468 2026-02-20T17:04:19.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.515468 2026-02-20T17:04:19.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.515468 2026-02-20T17:04:19.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.515468 2026-02-20T17:04:19.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.515468 2026-02-20T17:04:19.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.515468 2026-02-20T17:04:19.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.515468 2026-02-20T17:04:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.515468 2026-02-20T17:04:20.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.515468 2026-02-20T17:04:20.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.515468 2026-02-20T17:04:20.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.515468 2026-02-20T17:04:20.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.515468 2026-02-20T17:04:20.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.515468 2026-02-20T17:04:20.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.515468 2026-02-20T17:04:20.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.515468 2026-02-20T17:04:20.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.515468 2026-02-20T17:04:20.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.515468 2026-02-20T17:04:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.515468 2026-02-20T17:04:20.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.515468 2026-02-20T17:04:20.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.515468 2026-02-20T17:04:20.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.515468 2026-02-20T17:04:20.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.515468 2026-02-20T17:04:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.515468 2026-02-20T17:04:20.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.515468 2026-02-20T17:04:20.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.515468 2026-02-20T17:04:20.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.515468 2026-02-20T17:04:20.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.515468 2026-02-20T17:04:20.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.515468 2026-02-20T17:04:20.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.515468 2026-02-20T17:04:20.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.515468 2026-02-20T17:04:20.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.515468 2026-02-20T17:04:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.515468 2026-02-20T17:04:20.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.515468 2026-02-20T17:04:20.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.515468 2026-02-20T17:04:20.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.515468 2026-02-20T17:04:20.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.515468 2026-02-20T17:04:20.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.515468 2026-02-20T17:04:20.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.515468 2026-02-20T17:04:20.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.515468 2026-02-20T17:04:20.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.515468 2026-02-20T17:04:20.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.515468 2026-02-20T17:04:20.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.515468 2026-02-20T17:04:20.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.515468 2026-02-20T17:04:20.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.515468 2026-02-20T17:04:20.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.515468 2026-02-20T17:04:20.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.515468 2026-02-20T17:04:20.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.515468 2026-02-20T17:04:20.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.515468 2026-02-20T17:04:20.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.515468 2026-02-20T17:04:20.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.515468 2026-02-20T17:04:20.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:20.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.515468 2026-02-20T17:04:21.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.515468 2026-02-20T17:04:21.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.515468 2026-02-20T17:04:21.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.515468 2026-02-20T17:04:21.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.515468 2026-02-20T17:04:21.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.515468 2026-02-20T17:04:21.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.515468 2026-02-20T17:04:21.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.515468 2026-02-20T17:04:21.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.515468 2026-02-20T17:04:21.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.515468 2026-02-20T17:04:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.515468 2026-02-20T17:04:21.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.515468 2026-02-20T17:04:21.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.515468 2026-02-20T17:04:21.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.515468 2026-02-20T17:04:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.515468 2026-02-20T17:04:21.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.515468 2026-02-20T17:04:21.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.515468 2026-02-20T17:04:21.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.515468 2026-02-20T17:04:21.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.515468 2026-02-20T17:04:21.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.515468 2026-02-20T17:04:21.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.515468 2026-02-20T17:04:21.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.515468 2026-02-20T17:04:21.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.515468 2026-02-20T17:04:21.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.515468 2026-02-20T17:04:21.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.515468 2026-02-20T17:04:21.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.515468 2026-02-20T17:04:21.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.515468 2026-02-20T17:04:21.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.515468 2026-02-20T17:04:21.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.515468 2026-02-20T17:04:21.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.515468 2026-02-20T17:04:21.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.515468 2026-02-20T17:04:21.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.515468 2026-02-20T17:04:21.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.515468 2026-02-20T17:04:21.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.515468 2026-02-20T17:04:21.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.515468 2026-02-20T17:04:21.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.515468 2026-02-20T17:04:21.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.515468 2026-02-20T17:04:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.515468 2026-02-20T17:04:21.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.515468 2026-02-20T17:04:21.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.515468 2026-02-20T17:04:21.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.515468 2026-02-20T17:04:21.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.515468 2026-02-20T17:04:21.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.515468 2026-02-20T17:04:21.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.515468 2026-02-20T17:04:21.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:21.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.515468 2026-02-20T17:04:22.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.515468 2026-02-20T17:04:22.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.515468 2026-02-20T17:04:22.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.515468 2026-02-20T17:04:22.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.515468 2026-02-20T17:04:22.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.515468 2026-02-20T17:04:22.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.515468 2026-02-20T17:04:22.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.515468 2026-02-20T17:04:22.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.515468 2026-02-20T17:04:22.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.515468 2026-02-20T17:04:22.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.515468 2026-02-20T17:04:22.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.515468 2026-02-20T17:04:22.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.515468 2026-02-20T17:04:22.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.515468 2026-02-20T17:04:22.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.515468 2026-02-20T17:04:22.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.515468 2026-02-20T17:04:22.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.515468 2026-02-20T17:04:22.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.515468 2026-02-20T17:04:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.515468 2026-02-20T17:04:22.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.515468 2026-02-20T17:04:22.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.515468 2026-02-20T17:04:22.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.515468 2026-02-20T17:04:22.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.515468 2026-02-20T17:04:22.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.515468 2026-02-20T17:04:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.515468 2026-02-20T17:04:22.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.515468 2026-02-20T17:04:22.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.515468 2026-02-20T17:04:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.515468 2026-02-20T17:04:22.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.515468 2026-02-20T17:04:22.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.515468 2026-02-20T17:04:22.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.515468 2026-02-20T17:04:22.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.515468 2026-02-20T17:04:22.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.515468 2026-02-20T17:04:22.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.515468 2026-02-20T17:04:22.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.515468 2026-02-20T17:04:22.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.515468 2026-02-20T17:04:22.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.515468 2026-02-20T17:04:22.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.515468 2026-02-20T17:04:22.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.515468 2026-02-20T17:04:22.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.515468 2026-02-20T17:04:22.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.515468 2026-02-20T17:04:22.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.515468 2026-02-20T17:04:22.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.515468 2026-02-20T17:04:22.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.515468 2026-02-20T17:04:22.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:22.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.515468 2026-02-20T17:04:23.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.515468 2026-02-20T17:04:23.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.515468 2026-02-20T17:04:23.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.515468 2026-02-20T17:04:23.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.515468 2026-02-20T17:04:23.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.515468 2026-02-20T17:04:23.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.515468 2026-02-20T17:04:23.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.515468 2026-02-20T17:04:23.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.515468 2026-02-20T17:04:23.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.515468 2026-02-20T17:04:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.515468 2026-02-20T17:04:23.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.515468 2026-02-20T17:04:23.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.515468 2026-02-20T17:04:23.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.515468 2026-02-20T17:04:23.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.515468 2026-02-20T17:04:23.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.515468 2026-02-20T17:04:23.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.515468 2026-02-20T17:04:23.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.515468 2026-02-20T17:04:23.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.515468 2026-02-20T17:04:23.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.515468 2026-02-20T17:04:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.515468 2026-02-20T17:04:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.515468 2026-02-20T17:04:23.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.515468 2026-02-20T17:04:23.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.515468 2026-02-20T17:04:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.515468 2026-02-20T17:04:23.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.515468 2026-02-20T17:04:23.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.515468 2026-02-20T17:04:23.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.515468 2026-02-20T17:04:23.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.515468 2026-02-20T17:04:23.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.515468 2026-02-20T17:04:23.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.515468 2026-02-20T17:04:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.515468 2026-02-20T17:04:23.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.515468 2026-02-20T17:04:23.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.515468 2026-02-20T17:04:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.515468 2026-02-20T17:04:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.515468 2026-02-20T17:04:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.515468 2026-02-20T17:04:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.515468 2026-02-20T17:04:23.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.515468 2026-02-20T17:04:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.515468 2026-02-20T17:04:23.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.515468 2026-02-20T17:04:23.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.515468 2026-02-20T17:04:23.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.515468 2026-02-20T17:04:23.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.515468 2026-02-20T17:04:23.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.515468 2026-02-20T17:04:23.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.515468 2026-02-20T17:04:23.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:23.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.515468 2026-02-20T17:04:24.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.515468 2026-02-20T17:04:24.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.515468 2026-02-20T17:04:24.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.515468 2026-02-20T17:04:24.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.515468 2026-02-20T17:04:24.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.515468 2026-02-20T17:04:24.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.515468 2026-02-20T17:04:24.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.515468 2026-02-20T17:04:24.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.515468 2026-02-20T17:04:24.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.515468 2026-02-20T17:04:24.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.515468 2026-02-20T17:04:24.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.515468 2026-02-20T17:04:24.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.515468 2026-02-20T17:04:24.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.515468 2026-02-20T17:04:24.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.515468 2026-02-20T17:04:24.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.515468 2026-02-20T17:04:24.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.515468 2026-02-20T17:04:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.515468 2026-02-20T17:04:24.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.515468 2026-02-20T17:04:24.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.515468 2026-02-20T17:04:24.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.515468 2026-02-20T17:04:24.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.515468 2026-02-20T17:04:24.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.515468 2026-02-20T17:04:24.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.515468 2026-02-20T17:04:24.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.515468 2026-02-20T17:04:24.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.515468 2026-02-20T17:04:24.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.515468 2026-02-20T17:04:24.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.515468 2026-02-20T17:04:24.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.515468 2026-02-20T17:04:24.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.515468 2026-02-20T17:04:24.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.515468 2026-02-20T17:04:24.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.515468 2026-02-20T17:04:24.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.515468 2026-02-20T17:04:24.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.515468 2026-02-20T17:04:24.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.515468 2026-02-20T17:04:24.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.515468 2026-02-20T17:04:24.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.515468 2026-02-20T17:04:24.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.515468 2026-02-20T17:04:24.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.515468 2026-02-20T17:04:24.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.515468 2026-02-20T17:04:24.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.515468 2026-02-20T17:04:24.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.515468 2026-02-20T17:04:24.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.515468 2026-02-20T17:04:24.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.515468 2026-02-20T17:04:24.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:24.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.515468 2026-02-20T17:04:25.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.515468 2026-02-20T17:04:25.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.515468 2026-02-20T17:04:25.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.515468 2026-02-20T17:04:25.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.515468 2026-02-20T17:04:25.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.515468 2026-02-20T17:04:25.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.515468 2026-02-20T17:04:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.515468 2026-02-20T17:04:25.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.515468 2026-02-20T17:04:25.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.515468 2026-02-20T17:04:25.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.515468 2026-02-20T17:04:25.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.515468 2026-02-20T17:04:25.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.515468 2026-02-20T17:04:25.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.515468 2026-02-20T17:04:25.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.515468 2026-02-20T17:04:25.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.515468 2026-02-20T17:04:25.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.515468 2026-02-20T17:04:25.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.515468 2026-02-20T17:04:25.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.515468 2026-02-20T17:04:25.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.515468 2026-02-20T17:04:25.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.515468 2026-02-20T17:04:25.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.515468 2026-02-20T17:04:25.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.515468 2026-02-20T17:04:25.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.515468 2026-02-20T17:04:25.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.515468 2026-02-20T17:04:25.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.515468 2026-02-20T17:04:25.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.515468 2026-02-20T17:04:25.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.515468 2026-02-20T17:04:25.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.515468 2026-02-20T17:04:25.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.515468 2026-02-20T17:04:25.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.515468 2026-02-20T17:04:25.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.515468 2026-02-20T17:04:25.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.515468 2026-02-20T17:04:25.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.515468 2026-02-20T17:04:25.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.515468 2026-02-20T17:04:25.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.515468 2026-02-20T17:04:25.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.515468 2026-02-20T17:04:25.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.515468 2026-02-20T17:04:25.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.515468 2026-02-20T17:04:25.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.515468 2026-02-20T17:04:25.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.515468 2026-02-20T17:04:25.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.515468 2026-02-20T17:04:25.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:25.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.515468 2026-02-20T17:04:26.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.515468 2026-02-20T17:04:26.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.515468 2026-02-20T17:04:26.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.515468 2026-02-20T17:04:26.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.515468 2026-02-20T17:04:26.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.515468 2026-02-20T17:04:26.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.515468 2026-02-20T17:04:26.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.515468 2026-02-20T17:04:26.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.515468 2026-02-20T17:04:26.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.515468 2026-02-20T17:04:26.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.515468 2026-02-20T17:04:26.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.515468 2026-02-20T17:04:26.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.515468 2026-02-20T17:04:26.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.515468 2026-02-20T17:04:26.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.515468 2026-02-20T17:04:26.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.515468 2026-02-20T17:04:26.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.515468 2026-02-20T17:04:26.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.515468 2026-02-20T17:04:26.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.515468 2026-02-20T17:04:26.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.515468 2026-02-20T17:04:26.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.515468 2026-02-20T17:04:26.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.515468 2026-02-20T17:04:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.515468 2026-02-20T17:04:26.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.515468 2026-02-20T17:04:26.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.515468 2026-02-20T17:04:26.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.515468 2026-02-20T17:04:26.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.515468 2026-02-20T17:04:26.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.515468 2026-02-20T17:04:26.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.515468 2026-02-20T17:04:26.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.515468 2026-02-20T17:04:26.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.515468 2026-02-20T17:04:26.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.515468 2026-02-20T17:04:26.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.515468 2026-02-20T17:04:26.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.515468 2026-02-20T17:04:26.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.515468 2026-02-20T17:04:26.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.515468 2026-02-20T17:04:26.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.515468 2026-02-20T17:04:26.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.515468 2026-02-20T17:04:26.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.515468 2026-02-20T17:04:26.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.515468 2026-02-20T17:04:26.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.515468 2026-02-20T17:04:26.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.515468 2026-02-20T17:04:26.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:26.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.515468 2026-02-20T17:04:27.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.515468 2026-02-20T17:04:27.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.515468 2026-02-20T17:04:27.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.515468 2026-02-20T17:04:27.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.515468 2026-02-20T17:04:27.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.515468 2026-02-20T17:04:27.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.515468 2026-02-20T17:04:27.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.515468 2026-02-20T17:04:27.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.515468 2026-02-20T17:04:27.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.515468 2026-02-20T17:04:27.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.515468 2026-02-20T17:04:27.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.515468 2026-02-20T17:04:27.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.515468 2026-02-20T17:04:27.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.515468 2026-02-20T17:04:27.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.515468 2026-02-20T17:04:27.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.515468 2026-02-20T17:04:27.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.515468 2026-02-20T17:04:27.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.515468 2026-02-20T17:04:27.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.515468 2026-02-20T17:04:27.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.515468 2026-02-20T17:04:27.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.515468 2026-02-20T17:04:27.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.515468 2026-02-20T17:04:27.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.515468 2026-02-20T17:04:27.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.515468 2026-02-20T17:04:27.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.515468 2026-02-20T17:04:27.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.515468 2026-02-20T17:04:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.515468 2026-02-20T17:04:27.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.515468 2026-02-20T17:04:27.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.515468 2026-02-20T17:04:27.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.515468 2026-02-20T17:04:27.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.515468 2026-02-20T17:04:27.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.515468 2026-02-20T17:04:27.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.515468 2026-02-20T17:04:27.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.515468 2026-02-20T17:04:27.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.515468 2026-02-20T17:04:27.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.515468 2026-02-20T17:04:27.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.515468 2026-02-20T17:04:27.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.515468 2026-02-20T17:04:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.515468 2026-02-20T17:04:27.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.515468 2026-02-20T17:04:27.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.515468 2026-02-20T17:04:27.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.515468 2026-02-20T17:04:27.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.515468 2026-02-20T17:04:27.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.515468 2026-02-20T17:04:27.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:27.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.515468 2026-02-20T17:04:28.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.515468 2026-02-20T17:04:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.515468 2026-02-20T17:04:28.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.515468 2026-02-20T17:04:28.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.515468 2026-02-20T17:04:28.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.515468 2026-02-20T17:04:28.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.515468 2026-02-20T17:04:28.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.515468 2026-02-20T17:04:28.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.515468 2026-02-20T17:04:28.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.515468 2026-02-20T17:04:28.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.515468 2026-02-20T17:04:28.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.515468 2026-02-20T17:04:28.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.515468 2026-02-20T17:04:28.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.515468 2026-02-20T17:04:28.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.515468 2026-02-20T17:04:28.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.515468 2026-02-20T17:04:28.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.515468 2026-02-20T17:04:28.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.515468 2026-02-20T17:04:28.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.515468 2026-02-20T17:04:28.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.515468 2026-02-20T17:04:28.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.515468 2026-02-20T17:04:28.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.515468 2026-02-20T17:04:28.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.515468 2026-02-20T17:04:28.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.515468 2026-02-20T17:04:28.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.515468 2026-02-20T17:04:28.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.515468 2026-02-20T17:04:28.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.515468 2026-02-20T17:04:28.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.515468 2026-02-20T17:04:28.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.515468 2026-02-20T17:04:28.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.515468 2026-02-20T17:04:28.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.515468 2026-02-20T17:04:28.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.515468 2026-02-20T17:04:28.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.515468 2026-02-20T17:04:28.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.515468 2026-02-20T17:04:28.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.515468 2026-02-20T17:04:28.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.515468 2026-02-20T17:04:28.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.515468 2026-02-20T17:04:28.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.515468 2026-02-20T17:04:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.515468 2026-02-20T17:04:28.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.515468 2026-02-20T17:04:28.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.515468 2026-02-20T17:04:28.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:28.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.515468 2026-02-20T17:04:29.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.515468 2026-02-20T17:04:29.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.515468 2026-02-20T17:04:29.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.515468 2026-02-20T17:04:29.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.515468 2026-02-20T17:04:29.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.515468 2026-02-20T17:04:29.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.515468 2026-02-20T17:04:29.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.515468 2026-02-20T17:04:29.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.183 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.515468 2026-02-20T17:04:29.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.515468 2026-02-20T17:04:29.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.515468 2026-02-20T17:04:29.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.515468 2026-02-20T17:04:29.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.515468 2026-02-20T17:04:29.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.515468 2026-02-20T17:04:29.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.515468 2026-02-20T17:04:29.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.515468 2026-02-20T17:04:29.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.515468 2026-02-20T17:04:29.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.515468 2026-02-20T17:04:29.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.515468 2026-02-20T17:04:29.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.515468 2026-02-20T17:04:29.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.515468 2026-02-20T17:04:29.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.515468 2026-02-20T17:04:29.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.515468 2026-02-20T17:04:29.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.515468 2026-02-20T17:04:29.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.515468 2026-02-20T17:04:29.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.515468 2026-02-20T17:04:29.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.515468 2026-02-20T17:04:29.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.515468 2026-02-20T17:04:29.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.515468 2026-02-20T17:04:29.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.515468 2026-02-20T17:04:29.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.515468 2026-02-20T17:04:29.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.515468 2026-02-20T17:04:29.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.515468 2026-02-20T17:04:29.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.515468 2026-02-20T17:04:29.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.515468 2026-02-20T17:04:29.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.515468 2026-02-20T17:04:29.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.515468 2026-02-20T17:04:29.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.515468 2026-02-20T17:04:29.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.515468 2026-02-20T17:04:29.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.515468 2026-02-20T17:04:29.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.515468 2026-02-20T17:04:29.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.515468 2026-02-20T17:04:29.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.515468 2026-02-20T17:04:29.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:29.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.515468 2026-02-20T17:04:30.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.515468 2026-02-20T17:04:30.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.515468 2026-02-20T17:04:30.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.515468 2026-02-20T17:04:30.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.515468 2026-02-20T17:04:30.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.515468 2026-02-20T17:04:30.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.515468 2026-02-20T17:04:30.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.515468 2026-02-20T17:04:30.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.515468 2026-02-20T17:04:30.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.515468 2026-02-20T17:04:30.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:04:30.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.515468 2026-02-20T17:04:30.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.515468 2026-02-20T17:04:30.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-20T17:04:30.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:04:30.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:04:30.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:04:30.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:04:30.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-20T17:04:30.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-20T17:04:30.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2026-02-20T17:04:30.448 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:04:30.448 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T17:04:30.448 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:04:30.448 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T12:16:20.448005+0000" 2026-02-20T17:04:30.448 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:04:30.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-20T17:04:30.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-20T17:04:30.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-20T17:04:30.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:30.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-20T17:04:30.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:30.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:30.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:30.620 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:30.620 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:30.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:30.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:30.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:30.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836516 2026-02-20T17:04:30.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836516 2026-02-20T17:04:30.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516' 2026-02-20T17:04:30.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:30.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:30.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672993 2026-02-20T17:04:30.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672993 2026-02-20T17:04:30.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516 1-42949672993' 2026-02-20T17:04:30.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:30.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:30.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509471 2026-02-20T17:04:30.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509471 2026-02-20T17:04:30.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516 1-42949672993 2-64424509471' 2026-02-20T17:04:30.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:30.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836516 2026-02-20T17:04:30.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:30.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:30.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836516 2026-02-20T17:04:30.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:30.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836516 2026-02-20T17:04:30.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2026-02-20T17:04:30.841 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836516 2026-02-20T17:04:30.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:31.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836516 2026-02-20T17:04:31.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:32.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:32.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:32.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836516 2026-02-20T17:04:32.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:32.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672993 2026-02-20T17:04:32.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:32.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672993 2026-02-20T17:04:32.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:32.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672993 2026-02-20T17:04:32.170 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672993 2026-02-20T17:04:32.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2026-02-20T17:04:32.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:32.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672993 2026-02-20T17:04:32.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:32.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509471 2026-02-20T17:04:32.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:32.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:32.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509471 2026-02-20T17:04:32.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:32.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509471 2026-02-20T17:04:32.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2026-02-20T17:04:32.333 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509471 2026-02-20T17:04:32.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:32.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509472 -lt 64424509471 2026-02-20T17:04:32.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:32.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-20T17:04:32.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:32.631 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:32.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:32.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-20T17:04:32.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:32.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:32.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:32.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-20T17:04:32.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-20T17:04:32.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520' 2026-02-20T17:04:32.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:32.875 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:32.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-20T17:04:32.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-20T17:04:32.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-42949672997' 2026-02-20T17:04:32.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:32.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:33.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509474 2026-02-20T17:04:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509474 2026-02-20T17:04:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-42949672997 2-64424509474' 2026-02-20T17:04:33.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:33.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-20T17:04:33.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:33.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:33.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836520 2026-02-20T17:04:33.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:33.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836520 2026-02-20T17:04:33.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-20T17:04:33.026 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836520 2026-02-20T17:04:33.026 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:33.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836520 2026-02-20T17:04:33.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:34.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:34.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:34.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836520 -lt 21474836520 2026-02-20T17:04:34.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:34.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-20T17:04:34.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:34.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:34.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-20T17:04:34.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:34.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-20T17:04:34.352 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672997 2026-02-20T17:04:34.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-20T17:04:34.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:34.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672997 -lt 42949672997 2026-02-20T17:04:34.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:34.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509474 2026-02-20T17:04:34.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:34.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:34.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509474 2026-02-20T17:04:34.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:34.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509474 2026-02-20T17:04:34.520 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509474 2026-02-20T17:04:34.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2026-02-20T17:04:34.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:34.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509475 -lt 64424509474 2026-02-20T17:04:34.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:34.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-20T17:04:34.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:34.832 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-20T17:04:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-20T17:04:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-20T17:04:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-20T17:04:34.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set nodeep-scrub 2026-02-20T17:04:35.042 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:04:35.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-20T17:04:35.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:365: _scrub_abort: ceph osd set noscrub 2026-02-20T17:04:35.246 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:04:35.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-20T17:04:35.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-20T17:04:35.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:35.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:35.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:35.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:35.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:35.419 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:35.419 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:35.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:35.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:35.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836523 2026-02-20T17:04:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836523 2026-02-20T17:04:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836523' 2026-02-20T17:04:35.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:35.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:35.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673000 2026-02-20T17:04:35.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673000 2026-02-20T17:04:35.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836523 1-42949673000' 2026-02-20T17:04:35.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:35.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:35.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509478 2026-02-20T17:04:35.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509478 2026-02-20T17:04:35.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836523 1-42949673000 2-64424509478' 2026-02-20T17:04:35.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:35.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836523 2026-02-20T17:04:35.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:35.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:35.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836523 2026-02-20T17:04:35.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:35.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836523 2026-02-20T17:04:35.640 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836523 2026-02-20T17:04:35.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2026-02-20T17:04:35.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836523 2026-02-20T17:04:35.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:36.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:36.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:36.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836523 2026-02-20T17:04:36.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:37.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:04:37.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:38.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836523 2026-02-20T17:04:38.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:38.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673000 2026-02-20T17:04:38.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:38.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:38.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673000 2026-02-20T17:04:38.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:38.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673000 2026-02-20T17:04:38.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2026-02-20T17:04:38.123 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673000 2026-02-20T17:04:38.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673002 -lt 42949673000 2026-02-20T17:04:38.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:38.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509478 2026-02-20T17:04:38.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:38.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:38.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509478 2026-02-20T17:04:38.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:38.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509478 2026-02-20T17:04:38.296 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509478 2026-02-20T17:04:38.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2026-02-20T17:04:38.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:38.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509480 -lt 64424509478 2026-02-20T17:04:38.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:38.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:38.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:38.607 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:38.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:38.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:38.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:38.775 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:38.775 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:38.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:38.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:38.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:38.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836528 2026-02-20T17:04:38.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836528 2026-02-20T17:04:38.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836528' 2026-02-20T17:04:38.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:38.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:38.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673005 2026-02-20T17:04:38.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673005 2026-02-20T17:04:38.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836528 1-42949673005' 2026-02-20T17:04:38.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:38.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:38.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509482 2026-02-20T17:04:38.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509482 2026-02-20T17:04:38.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836528 1-42949673005 2-64424509482' 2026-02-20T17:04:38.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:38.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836528 2026-02-20T17:04:38.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:38.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:38.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836528 2026-02-20T17:04:38.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:38.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836528 2026-02-20T17:04:38.997 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836528 2026-02-20T17:04:38.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2026-02-20T17:04:38.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:39.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836528 2026-02-20T17:04:39.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:40.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:40.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836528 -lt 21474836528 2026-02-20T17:04:40.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:40.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673005 2026-02-20T17:04:40.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:40.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:40.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673005 2026-02-20T17:04:40.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:40.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673005 2026-02-20T17:04:40.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2026-02-20T17:04:40.313 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673005 2026-02-20T17:04:40.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:40.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673005 -lt 42949673005 2026-02-20T17:04:40.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:40.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509482 2026-02-20T17:04:40.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:40.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:40.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509482 2026-02-20T17:04:40.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:40.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509482 2026-02-20T17:04:40.470 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509482 2026-02-20T17:04:40.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2026-02-20T17:04:40.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:40.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509483 -lt 64424509482 2026-02-20T17:04:40.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:40.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:40.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:40.772 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:40.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:40.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:40.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:40.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:40.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:40.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:40.941 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:40.941 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:40.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:40.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:40.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:41.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836531 2026-02-20T17:04:41.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836531 2026-02-20T17:04:41.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531' 2026-02-20T17:04:41.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:41.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:41.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673008 2026-02-20T17:04:41.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673008 2026-02-20T17:04:41.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531 1-42949673008' 2026-02-20T17:04:41.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:41.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:41.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509485 2026-02-20T17:04:41.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509485 2026-02-20T17:04:41.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531 1-42949673008 2-64424509485' 2026-02-20T17:04:41.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:41.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836531 2026-02-20T17:04:41.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:41.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:41.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836531 2026-02-20T17:04:41.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:41.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836531 2026-02-20T17:04:41.166 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836531 2026-02-20T17:04:41.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2026-02-20T17:04:41.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:41.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836528 -lt 21474836531 2026-02-20T17:04:41.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:42.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:42.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:42.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836531 -lt 21474836531 2026-02-20T17:04:42.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:42.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673008 2026-02-20T17:04:42.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:42.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:42.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673008 2026-02-20T17:04:42.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:42.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673008 2026-02-20T17:04:42.493 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673008 2026-02-20T17:04:42.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2026-02-20T17:04:42.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:42.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673008 -lt 42949673008 2026-02-20T17:04:42.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:42.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509485 2026-02-20T17:04:42.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:42.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:42.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509485 2026-02-20T17:04:42.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:42.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509485 2026-02-20T17:04:42.662 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509485 2026-02-20T17:04:42.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2026-02-20T17:04:42.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:42.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509486 -lt 64424509485 2026-02-20T17:04:42.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:42.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:42.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:42.977 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:42.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:42.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:42.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:42.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:42.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:43.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:43.140 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:43.140 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:43.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:43.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:43.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:43.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836534 2026-02-20T17:04:43.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836534 2026-02-20T17:04:43.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836534' 2026-02-20T17:04:43.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:43.217 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673011 2026-02-20T17:04:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673011 2026-02-20T17:04:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836534 1-42949673011' 2026-02-20T17:04:43.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:43.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:43.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509489 2026-02-20T17:04:43.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509489 2026-02-20T17:04:43.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836534 1-42949673011 2-64424509489' 2026-02-20T17:04:43.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:43.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836534 2026-02-20T17:04:43.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:43.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:43.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836534 2026-02-20T17:04:43.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:43.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836534 2026-02-20T17:04:43.364 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836534 2026-02-20T17:04:43.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2026-02-20T17:04:43.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:43.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836534 -lt 21474836534 2026-02-20T17:04:43.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:43.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673011 2026-02-20T17:04:43.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:43.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:43.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673011 2026-02-20T17:04:43.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:43.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673011 2026-02-20T17:04:43.528 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673011 2026-02-20T17:04:43.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2026-02-20T17:04:43.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:43.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673011 -lt 42949673011 2026-02-20T17:04:43.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:43.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509489 2026-02-20T17:04:43.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:43.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:43.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509489 2026-02-20T17:04:43.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:43.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509489 2026-02-20T17:04:43.689 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509489 2026-02-20T17:04:43.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2026-02-20T17:04:43.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:43.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509489 -lt 64424509489 2026-02-20T17:04:43.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:43.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:43.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:44.005 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:44.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:44.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:44.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:44.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:44.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:44.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:44.170 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:44.170 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:44.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:44.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:44.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836536 2026-02-20T17:04:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836536 2026-02-20T17:04:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536' 2026-02-20T17:04:44.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:44.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:44.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673013 2026-02-20T17:04:44.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673013 2026-02-20T17:04:44.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-42949673013' 2026-02-20T17:04:44.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:44.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:44.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509491 2026-02-20T17:04:44.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509491 2026-02-20T17:04:44.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-42949673013 2-64424509491' 2026-02-20T17:04:44.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:44.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836536 2026-02-20T17:04:44.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:44.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:44.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836536 2026-02-20T17:04:44.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:44.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836536 2026-02-20T17:04:44.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2026-02-20T17:04:44.393 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836536 2026-02-20T17:04:44.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836534 -lt 21474836536 2026-02-20T17:04:44.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:45.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:45.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836537 -lt 21474836536 2026-02-20T17:04:45.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:45.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673013 2026-02-20T17:04:45.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:45.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:45.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673013 2026-02-20T17:04:45.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:45.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673013 2026-02-20T17:04:45.727 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673013 2026-02-20T17:04:45.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2026-02-20T17:04:45.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:45.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673014 -lt 42949673013 2026-02-20T17:04:45.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:45.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509491 2026-02-20T17:04:45.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:45.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:45.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509491 2026-02-20T17:04:45.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:45.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509491 2026-02-20T17:04:45.889 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509491 2026-02-20T17:04:45.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2026-02-20T17:04:45.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:46.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509492 -lt 64424509491 2026-02-20T17:04:46.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:46.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:46.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:46.199 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:46.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:46.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:46.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:46.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:46.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:46.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:46.364 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:46.364 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:46.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:46.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:46.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:46.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836539 2026-02-20T17:04:46.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836539 2026-02-20T17:04:46.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539' 2026-02-20T17:04:46.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:46.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:46.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673016 2026-02-20T17:04:46.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673016 2026-02-20T17:04:46.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-42949673016' 2026-02-20T17:04:46.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:46.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509494 2026-02-20T17:04:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509494 2026-02-20T17:04:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-42949673016 2-64424509494' 2026-02-20T17:04:46.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:46.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836539 2026-02-20T17:04:46.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:46.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:46.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836539 2026-02-20T17:04:46.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:46.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836539 2026-02-20T17:04:46.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2026-02-20T17:04:46.585 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836539 2026-02-20T17:04:46.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:46.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836537 -lt 21474836539 2026-02-20T17:04:46.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:47.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:47.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:47.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836540 -lt 21474836539 2026-02-20T17:04:47.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:47.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673016 2026-02-20T17:04:47.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:47.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:47.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673016 2026-02-20T17:04:47.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:47.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673016 2026-02-20T17:04:47.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2026-02-20T17:04:47.909 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673016 2026-02-20T17:04:47.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:48.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673017 -lt 42949673016 2026-02-20T17:04:48.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:48.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509494 2026-02-20T17:04:48.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:48.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:48.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509494 2026-02-20T17:04:48.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:48.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509494 2026-02-20T17:04:48.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2026-02-20T17:04:48.071 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509494 2026-02-20T17:04:48.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509495 -lt 64424509494 2026-02-20T17:04:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:48.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:48.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:48.385 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:48.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:48.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:48.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:48.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:48.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:48.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:48.557 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:48.557 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:48.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:48.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:48.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:48.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836542 2026-02-20T17:04:48.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836542 2026-02-20T17:04:48.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542' 2026-02-20T17:04:48.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:48.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:48.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673019 2026-02-20T17:04:48.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673019 2026-02-20T17:04:48.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542 1-42949673019' 2026-02-20T17:04:48.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:48.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509497 2026-02-20T17:04:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509497 2026-02-20T17:04:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542 1-42949673019 2-64424509497' 2026-02-20T17:04:48.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:48.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836542 2026-02-20T17:04:48.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:48.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:48.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836542 2026-02-20T17:04:48.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:48.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836542 2026-02-20T17:04:48.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2026-02-20T17:04:48.784 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836542 2026-02-20T17:04:48.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:48.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836540 -lt 21474836542 2026-02-20T17:04:48.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:49.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:49.945 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:50.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836543 -lt 21474836542 2026-02-20T17:04:50.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673019 2026-02-20T17:04:50.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:50.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:50.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673019 2026-02-20T17:04:50.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:50.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673019 2026-02-20T17:04:50.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2026-02-20T17:04:50.104 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673019 2026-02-20T17:04:50.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:50.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673020 -lt 42949673019 2026-02-20T17:04:50.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:50.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509497 2026-02-20T17:04:50.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:50.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:50.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509497 2026-02-20T17:04:50.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:50.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509497 2026-02-20T17:04:50.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2026-02-20T17:04:50.279 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509497 2026-02-20T17:04:50.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:50.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509498 -lt 64424509497 2026-02-20T17:04:50.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:50.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:50.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:50.595 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:50.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:50.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:50.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:50.763 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:50.763 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:50.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:50.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:50.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:50.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836546 2026-02-20T17:04:50.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836546 2026-02-20T17:04:50.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546' 2026-02-20T17:04:50.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:50.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:50.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673023 2026-02-20T17:04:50.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673023 2026-02-20T17:04:50.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546 1-42949673023' 2026-02-20T17:04:50.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:50.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:50.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509500 2026-02-20T17:04:50.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509500 2026-02-20T17:04:50.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546 1-42949673023 2-64424509500' 2026-02-20T17:04:50.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:50.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836546 2026-02-20T17:04:50.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:50.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:50.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836546 2026-02-20T17:04:50.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:50.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836546 2026-02-20T17:04:50.982 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836546 2026-02-20T17:04:50.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2026-02-20T17:04:50.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:51.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836543 -lt 21474836546 2026-02-20T17:04:51.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:52.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:52.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:52.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836546 -lt 21474836546 2026-02-20T17:04:52.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:52.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673023 2026-02-20T17:04:52.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:52.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:52.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673023 2026-02-20T17:04:52.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:52.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673023 2026-02-20T17:04:52.315 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673023 2026-02-20T17:04:52.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2026-02-20T17:04:52.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:52.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673023 -lt 42949673023 2026-02-20T17:04:52.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:52.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509500 2026-02-20T17:04:52.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:52.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:52.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509500 2026-02-20T17:04:52.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:52.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509500 2026-02-20T17:04:52.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2026-02-20T17:04:52.477 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509500 2026-02-20T17:04:52.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:52.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509501 -lt 64424509500 2026-02-20T17:04:52.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:52.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:52.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:52.782 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:52.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:52.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:52.951 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:52.951 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:52.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:52.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:52.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:53.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836549 2026-02-20T17:04:53.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836549 2026-02-20T17:04:53.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836549' 2026-02-20T17:04:53.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:53.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:53.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673026 2026-02-20T17:04:53.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673026 2026-02-20T17:04:53.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836549 1-42949673026' 2026-02-20T17:04:53.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:53.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:53.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509503 2026-02-20T17:04:53.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509503 2026-02-20T17:04:53.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836549 1-42949673026 2-64424509503' 2026-02-20T17:04:53.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:53.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836549 2026-02-20T17:04:53.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:53.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:53.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836549 2026-02-20T17:04:53.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:53.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836549 2026-02-20T17:04:53.170 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836549 2026-02-20T17:04:53.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2026-02-20T17:04:53.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:53.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836546 -lt 21474836549 2026-02-20T17:04:53.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:54.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:54.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:54.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836549 -lt 21474836549 2026-02-20T17:04:54.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:54.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673026 2026-02-20T17:04:54.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:54.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:54.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673026 2026-02-20T17:04:54.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:54.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673026 2026-02-20T17:04:54.495 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673026 2026-02-20T17:04:54.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2026-02-20T17:04:54.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:54.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673026 -lt 42949673026 2026-02-20T17:04:54.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:54.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509503 2026-02-20T17:04:54.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:54.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:54.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509503 2026-02-20T17:04:54.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:54.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509503 2026-02-20T17:04:54.666 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509503 2026-02-20T17:04:54.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2026-02-20T17:04:54.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:54.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509504 -lt 64424509503 2026-02-20T17:04:54.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:54.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:54.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:54.982 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:54.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:54.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:55.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:55.148 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:55.148 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:55.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:55.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:55.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836552 2026-02-20T17:04:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836552 2026-02-20T17:04:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836552' 2026-02-20T17:04:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:55.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:55.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673029 2026-02-20T17:04:55.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673029 2026-02-20T17:04:55.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836552 1-42949673029' 2026-02-20T17:04:55.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:55.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:55.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509507 2026-02-20T17:04:55.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509507 2026-02-20T17:04:55.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836552 1-42949673029 2-64424509507' 2026-02-20T17:04:55.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:55.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836552 2026-02-20T17:04:55.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:55.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:55.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836552 2026-02-20T17:04:55.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:55.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836552 2026-02-20T17:04:55.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2026-02-20T17:04:55.366 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836552 2026-02-20T17:04:55.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:55.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836552 -lt 21474836552 2026-02-20T17:04:55.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:55.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673029 2026-02-20T17:04:55.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:55.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:55.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673029 2026-02-20T17:04:55.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673029 2026-02-20T17:04:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2026-02-20T17:04:55.532 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673029 2026-02-20T17:04:55.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:55.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673029 -lt 42949673029 2026-02-20T17:04:55.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:55.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509507 2026-02-20T17:04:55.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:55.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:55.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509507 2026-02-20T17:04:55.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:55.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509507 2026-02-20T17:04:55.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2026-02-20T17:04:55.706 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509507 2026-02-20T17:04:55.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:55.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509507 -lt 64424509507 2026-02-20T17:04:55.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:55.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:55.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:56.022 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:56.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:56.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:56.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:56.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:56.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:56.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:56.187 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:56.187 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:56.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:56.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:56.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:56.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836554 2026-02-20T17:04:56.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836554 2026-02-20T17:04:56.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554' 2026-02-20T17:04:56.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:56.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:56.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673031 2026-02-20T17:04:56.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673031 2026-02-20T17:04:56.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-42949673031' 2026-02-20T17:04:56.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:56.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:56.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509509 2026-02-20T17:04:56.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509509 2026-02-20T17:04:56.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-42949673031 2-64424509509' 2026-02-20T17:04:56.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:56.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836554 2026-02-20T17:04:56.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:56.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:56.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836554 2026-02-20T17:04:56.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:56.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836554 2026-02-20T17:04:56.408 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836554 2026-02-20T17:04:56.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2026-02-20T17:04:56.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:56.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836552 -lt 21474836554 2026-02-20T17:04:56.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:57.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:57.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:57.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836555 -lt 21474836554 2026-02-20T17:04:57.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:57.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673031 2026-02-20T17:04:57.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:57.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:57.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673031 2026-02-20T17:04:57.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:57.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673031 2026-02-20T17:04:57.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2026-02-20T17:04:57.726 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673031 2026-02-20T17:04:57.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:04:57.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673032 -lt 42949673031 2026-02-20T17:04:57.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:57.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509509 2026-02-20T17:04:57.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:57.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:04:57.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509509 2026-02-20T17:04:57.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:57.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509509 2026-02-20T17:04:57.895 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509509 2026-02-20T17:04:57.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2026-02-20T17:04:57.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:04:58.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509510 -lt 64424509509 2026-02-20T17:04:58.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:04:58.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:04:58.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:04:58.210 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:04:58.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:04:58.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:04:58.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:04:58.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:04:58.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:04:58.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:04:58.376 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:04:58.376 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:04:58.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:04:58.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:58.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:04:58.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836557 2026-02-20T17:04:58.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836557 2026-02-20T17:04:58.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836557' 2026-02-20T17:04:58.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:58.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:04:58.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673034 2026-02-20T17:04:58.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673034 2026-02-20T17:04:58.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836557 1-42949673034' 2026-02-20T17:04:58.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:04:58.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:04:58.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509512 2026-02-20T17:04:58.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509512 2026-02-20T17:04:58.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836557 1-42949673034 2-64424509512' 2026-02-20T17:04:58.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836557 2026-02-20T17:04:58.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:58.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:04:58.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836557 2026-02-20T17:04:58.592 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836557 2026-02-20T17:04:58.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2026-02-20T17:04:58.594 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836557 2026-02-20T17:04:58.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:58.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836555 -lt 21474836557 2026-02-20T17:04:58.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:04:59.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:04:59.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:04:59.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836558 -lt 21474836557 2026-02-20T17:04:59.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:04:59.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673034 2026-02-20T17:04:59.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:04:59.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:04:59.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673034 2026-02-20T17:04:59.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:04:59.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673034 2026-02-20T17:04:59.923 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673034 2026-02-20T17:04:59.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2026-02-20T17:04:59.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:00.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673035 -lt 42949673034 2026-02-20T17:05:00.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:00.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509512 2026-02-20T17:05:00.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:00.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:00.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509512 2026-02-20T17:05:00.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:00.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509512 2026-02-20T17:05:00.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2026-02-20T17:05:00.086 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509512 2026-02-20T17:05:00.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:00.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509513 -lt 64424509512 2026-02-20T17:05:00.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:00.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:00.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:00.400 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:00.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:00.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:00.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:00.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:00.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:00.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:00.568 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:00.568 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:00.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:00.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:00.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:00.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836560 2026-02-20T17:05:00.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836560 2026-02-20T17:05:00.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836560' 2026-02-20T17:05:00.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:00.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:00.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673037 2026-02-20T17:05:00.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673037 2026-02-20T17:05:00.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836560 1-42949673037' 2026-02-20T17:05:00.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:00.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:00.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509515 2026-02-20T17:05:00.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509515 2026-02-20T17:05:00.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836560 1-42949673037 2-64424509515' 2026-02-20T17:05:00.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:00.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836560 2026-02-20T17:05:00.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:00.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:00.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836560 2026-02-20T17:05:00.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:00.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836560 2026-02-20T17:05:00.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2026-02-20T17:05:00.784 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836560 2026-02-20T17:05:00.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:00.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836558 -lt 21474836560 2026-02-20T17:05:00.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:01.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:01.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:02.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836561 -lt 21474836560 2026-02-20T17:05:02.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:02.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673037 2026-02-20T17:05:02.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:02.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:02.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673037 2026-02-20T17:05:02.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:02.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673037 2026-02-20T17:05:02.117 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673037 2026-02-20T17:05:02.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2026-02-20T17:05:02.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:02.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673038 -lt 42949673037 2026-02-20T17:05:02.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:02.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509515 2026-02-20T17:05:02.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:02.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:02.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509515 2026-02-20T17:05:02.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:02.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509515 2026-02-20T17:05:02.289 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509515 2026-02-20T17:05:02.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2026-02-20T17:05:02.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:02.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509516 -lt 64424509515 2026-02-20T17:05:02.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:02.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:02.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:02.592 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:02.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:02.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:02.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:02.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:02.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:02.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:02.763 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:02.764 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:02.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:02.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:02.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:02.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836564 2026-02-20T17:05:02.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836564 2026-02-20T17:05:02.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564' 2026-02-20T17:05:02.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:02.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:02.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673041 2026-02-20T17:05:02.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673041 2026-02-20T17:05:02.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-42949673041' 2026-02-20T17:05:02.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:02.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:02.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509518 2026-02-20T17:05:02.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509518 2026-02-20T17:05:02.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-42949673041 2-64424509518' 2026-02-20T17:05:02.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:02.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836564 2026-02-20T17:05:02.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:02.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:02.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836564 2026-02-20T17:05:02.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:02.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836564 2026-02-20T17:05:02.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2026-02-20T17:05:02.982 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836564 2026-02-20T17:05:02.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:03.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836561 -lt 21474836564 2026-02-20T17:05:03.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:04.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:04.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:04.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836564 -lt 21474836564 2026-02-20T17:05:04.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:04.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673041 2026-02-20T17:05:04.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:04.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:04.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673041 2026-02-20T17:05:04.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:04.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673041 2026-02-20T17:05:04.317 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673041 2026-02-20T17:05:04.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2026-02-20T17:05:04.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:04.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673041 -lt 42949673041 2026-02-20T17:05:04.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:04.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509518 2026-02-20T17:05:04.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:04.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:04.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509518 2026-02-20T17:05:04.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:04.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509518 2026-02-20T17:05:04.489 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509518 2026-02-20T17:05:04.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2026-02-20T17:05:04.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:04.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509519 -lt 64424509518 2026-02-20T17:05:04.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:04.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:04.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:04.805 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:04.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:04.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:04.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:04.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:04.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:04.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:04.976 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:04.976 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:04.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:04.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:04.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:05.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836567 2026-02-20T17:05:05.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836567 2026-02-20T17:05:05.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567' 2026-02-20T17:05:05.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:05.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673044 2026-02-20T17:05:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673044 2026-02-20T17:05:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-42949673044' 2026-02-20T17:05:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:05.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:05.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509521 2026-02-20T17:05:05.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509521 2026-02-20T17:05:05.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-42949673044 2-64424509521' 2026-02-20T17:05:05.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:05.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836567 2026-02-20T17:05:05.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:05.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:05.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836567 2026-02-20T17:05:05.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:05.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836567 2026-02-20T17:05:05.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2026-02-20T17:05:05.194 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836567 2026-02-20T17:05:05.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:05.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836564 -lt 21474836567 2026-02-20T17:05:05.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:06.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:06.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:06.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836567 -lt 21474836567 2026-02-20T17:05:06.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:06.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673044 2026-02-20T17:05:06.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:06.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:06.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673044 2026-02-20T17:05:06.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:06.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673044 2026-02-20T17:05:06.524 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673044 2026-02-20T17:05:06.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2026-02-20T17:05:06.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:06.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673044 -lt 42949673044 2026-02-20T17:05:06.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:06.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509521 2026-02-20T17:05:06.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:06.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:06.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509521 2026-02-20T17:05:06.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:06.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509521 2026-02-20T17:05:06.692 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509521 2026-02-20T17:05:06.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2026-02-20T17:05:06.693 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:06.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509522 -lt 64424509521 2026-02-20T17:05:06.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:06.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:06.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:06.993 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:07.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:07.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:07.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:07.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:07.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:07.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:07.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836570 2026-02-20T17:05:07.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836570 2026-02-20T17:05:07.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570' 2026-02-20T17:05:07.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:07.239 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673047 2026-02-20T17:05:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673047 2026-02-20T17:05:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-42949673047' 2026-02-20T17:05:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:07.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509525 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509525 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525' 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836570 2026-02-20T17:05:07.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:07.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:07.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836570 2026-02-20T17:05:07.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836570 2026-02-20T17:05:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2026-02-20T17:05:07.386 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836570 2026-02-20T17:05:07.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:07.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836570 -lt 21474836570 2026-02-20T17:05:07.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:07.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673047 2026-02-20T17:05:07.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:07.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:07.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673047 2026-02-20T17:05:07.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:07.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673047 2026-02-20T17:05:07.557 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673047 2026-02-20T17:05:07.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2026-02-20T17:05:07.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:07.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673047 -lt 42949673047 2026-02-20T17:05:07.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:07.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509525 2026-02-20T17:05:07.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:07.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:07.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509525 2026-02-20T17:05:07.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:07.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509525 2026-02-20T17:05:07.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2026-02-20T17:05:07.726 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509525 2026-02-20T17:05:07.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:07.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509525 -lt 64424509525 2026-02-20T17:05:07.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:07.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:07.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:08.043 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:08.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:08.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:08.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:08.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:08.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:08.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:08.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836572 2026-02-20T17:05:08.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836572 2026-02-20T17:05:08.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572' 2026-02-20T17:05:08.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:08.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:08.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673049 2026-02-20T17:05:08.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673049 2026-02-20T17:05:08.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572 1-42949673049' 2026-02-20T17:05:08.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:08.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509527 2026-02-20T17:05:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509527 2026-02-20T17:05:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509527' 2026-02-20T17:05:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:08.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836572 2026-02-20T17:05:08.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:08.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:08.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836572 2026-02-20T17:05:08.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:08.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836572 2026-02-20T17:05:08.426 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836572 2026-02-20T17:05:08.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2026-02-20T17:05:08.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:08.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836570 -lt 21474836572 2026-02-20T17:05:08.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:09.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:09.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:09.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836573 -lt 21474836572 2026-02-20T17:05:09.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:09.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673049 2026-02-20T17:05:09.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:09.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:09.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673049 2026-02-20T17:05:09.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:09.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673049 2026-02-20T17:05:09.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2026-02-20T17:05:09.758 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673049 2026-02-20T17:05:09.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:09.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673050 -lt 42949673049 2026-02-20T17:05:09.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:09.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509527 2026-02-20T17:05:09.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:09.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:09.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509527 2026-02-20T17:05:09.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:09.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509527 2026-02-20T17:05:09.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2026-02-20T17:05:09.925 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509527 2026-02-20T17:05:09.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:10.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509528 -lt 64424509527 2026-02-20T17:05:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:10.234 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:10.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:10.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:10.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:10.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:10.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:10.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:10.405 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:10.405 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:10.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:10.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:10.405 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:10.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836575 2026-02-20T17:05:10.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836575 2026-02-20T17:05:10.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575' 2026-02-20T17:05:10.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:10.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:10.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673052 2026-02-20T17:05:10.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673052 2026-02-20T17:05:10.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-42949673052' 2026-02-20T17:05:10.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:10.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:10.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509530 2026-02-20T17:05:10.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509530 2026-02-20T17:05:10.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509530' 2026-02-20T17:05:10.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:10.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836575 2026-02-20T17:05:10.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:10.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:10.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836575 2026-02-20T17:05:10.625 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836575 2026-02-20T17:05:10.626 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836575 2026-02-20T17:05:10.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2026-02-20T17:05:10.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836573 -lt 21474836575 2026-02-20T17:05:10.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:11.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:11.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:11.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836576 -lt 21474836575 2026-02-20T17:05:11.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:11.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673052 2026-02-20T17:05:11.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:11.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:11.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673052 2026-02-20T17:05:11.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:11.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673052 2026-02-20T17:05:11.960 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673052 2026-02-20T17:05:11.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2026-02-20T17:05:11.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:12.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673053 -lt 42949673052 2026-02-20T17:05:12.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:12.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509530 2026-02-20T17:05:12.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:12.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:12.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509530 2026-02-20T17:05:12.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:12.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509530 2026-02-20T17:05:12.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2026-02-20T17:05:12.118 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509530 2026-02-20T17:05:12.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:12.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509531 -lt 64424509530 2026-02-20T17:05:12.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:12.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:12.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:12.430 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:12.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:12.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:12.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:12.593 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:12.593 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:12.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:12.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:12.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:12.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836578 2026-02-20T17:05:12.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836578 2026-02-20T17:05:12.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578' 2026-02-20T17:05:12.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:12.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:12.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673055 2026-02-20T17:05:12.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673055 2026-02-20T17:05:12.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-42949673055' 2026-02-20T17:05:12.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:12.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:12.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509533 2026-02-20T17:05:12.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509533 2026-02-20T17:05:12.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-42949673055 2-64424509533' 2026-02-20T17:05:12.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:12.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836578 2026-02-20T17:05:12.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:12.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:12.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836578 2026-02-20T17:05:12.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:12.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836578 2026-02-20T17:05:12.816 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836578 2026-02-20T17:05:12.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2026-02-20T17:05:12.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836576 -lt 21474836578 2026-02-20T17:05:12.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:13.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:13.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:14.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836578 2026-02-20T17:05:14.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:14.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673055 2026-02-20T17:05:14.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:14.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:14.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673055 2026-02-20T17:05:14.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:14.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673055 2026-02-20T17:05:14.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2026-02-20T17:05:14.144 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673055 2026-02-20T17:05:14.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:14.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673056 -lt 42949673055 2026-02-20T17:05:14.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:14.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509533 2026-02-20T17:05:14.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:14.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:14.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509533 2026-02-20T17:05:14.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:14.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509533 2026-02-20T17:05:14.308 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509533 2026-02-20T17:05:14.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2026-02-20T17:05:14.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:14.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509534 -lt 64424509533 2026-02-20T17:05:14.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:14.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:14.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:14.618 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:14.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:14.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:14.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:14.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:14.628 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:14.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:14.784 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:14.784 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:14.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:14.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:14.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:14.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836582 2026-02-20T17:05:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836582 2026-02-20T17:05:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582' 2026-02-20T17:05:14.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:14.858 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:14.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673059 2026-02-20T17:05:14.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673059 2026-02-20T17:05:14.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-42949673059' 2026-02-20T17:05:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:14.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:15.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509536 2026-02-20T17:05:15.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509536 2026-02-20T17:05:15.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509536' 2026-02-20T17:05:15.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:15.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836582 2026-02-20T17:05:15.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:15.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:15.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836582 2026-02-20T17:05:15.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:15.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836582 2026-02-20T17:05:15.011 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836582 2026-02-20T17:05:15.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2026-02-20T17:05:15.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:15.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836582 2026-02-20T17:05:15.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:16.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:16.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:16.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836582 -lt 21474836582 2026-02-20T17:05:16.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:16.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673059 2026-02-20T17:05:16.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:16.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:16.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673059 2026-02-20T17:05:16.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:16.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673059 2026-02-20T17:05:16.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2026-02-20T17:05:16.339 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673059 2026-02-20T17:05:16.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:16.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673059 -lt 42949673059 2026-02-20T17:05:16.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:16.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509536 2026-02-20T17:05:16.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:16.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:16.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509536 2026-02-20T17:05:16.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:16.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509536 2026-02-20T17:05:16.515 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509536 2026-02-20T17:05:16.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2026-02-20T17:05:16.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:16.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509537 -lt 64424509536 2026-02-20T17:05:16.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:16.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:16.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:16.825 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:16.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:16.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:16.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:16.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:16.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:17.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:17.000 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:17.000 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:17.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:17.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:17.001 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836585 2026-02-20T17:05:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836585 2026-02-20T17:05:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585' 2026-02-20T17:05:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:17.075 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:17.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673062 2026-02-20T17:05:17.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673062 2026-02-20T17:05:17.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-42949673062' 2026-02-20T17:05:17.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:17.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:17.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509539 2026-02-20T17:05:17.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509539 2026-02-20T17:05:17.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509539' 2026-02-20T17:05:17.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:17.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836585 2026-02-20T17:05:17.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:17.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:17.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836585 2026-02-20T17:05:17.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:17.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836585 2026-02-20T17:05:17.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2026-02-20T17:05:17.221 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836585 2026-02-20T17:05:17.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:17.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836582 -lt 21474836585 2026-02-20T17:05:17.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:18.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:18.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:18.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836585 -lt 21474836585 2026-02-20T17:05:18.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:18.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673062 2026-02-20T17:05:18.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:18.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:18.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673062 2026-02-20T17:05:18.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:18.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673062 2026-02-20T17:05:18.552 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673062 2026-02-20T17:05:18.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2026-02-20T17:05:18.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:18.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673062 -lt 42949673062 2026-02-20T17:05:18.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:18.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509539 2026-02-20T17:05:18.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:18.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:18.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509539 2026-02-20T17:05:18.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:18.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509539 2026-02-20T17:05:18.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2026-02-20T17:05:18.729 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509539 2026-02-20T17:05:18.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:18.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509540 -lt 64424509539 2026-02-20T17:05:18.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:18.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:18.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:19.035 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:19.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:19.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:19.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:19.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:19.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:19.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:19.201 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:19.201 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:19.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:19.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:19.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:19.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836588 2026-02-20T17:05:19.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836588 2026-02-20T17:05:19.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588' 2026-02-20T17:05:19.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:19.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:19.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673065 2026-02-20T17:05:19.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673065 2026-02-20T17:05:19.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-42949673065' 2026-02-20T17:05:19.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:19.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:19.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509543 2026-02-20T17:05:19.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509543 2026-02-20T17:05:19.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-42949673065 2-64424509543' 2026-02-20T17:05:19.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:19.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836588 2026-02-20T17:05:19.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:19.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:19.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836588 2026-02-20T17:05:19.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:19.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836588 2026-02-20T17:05:19.423 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836588 2026-02-20T17:05:19.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2026-02-20T17:05:19.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:19.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836588 -lt 21474836588 2026-02-20T17:05:19.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:19.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673065 2026-02-20T17:05:19.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:19.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:19.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673065 2026-02-20T17:05:19.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:19.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673065 2026-02-20T17:05:19.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2026-02-20T17:05:19.597 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673065 2026-02-20T17:05:19.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:19.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673065 -lt 42949673065 2026-02-20T17:05:19.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:19.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509543 2026-02-20T17:05:19.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:19.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:19.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509543 2026-02-20T17:05:19.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:19.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509543 2026-02-20T17:05:19.767 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509543 2026-02-20T17:05:19.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2026-02-20T17:05:19.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:19.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509542 -lt 64424509543 2026-02-20T17:05:19.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:20.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:20.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:21.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509542 -lt 64424509543 2026-02-20T17:05:21.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:22.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:05:22.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:22.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509545 -lt 64424509543 2026-02-20T17:05:22.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:22.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:22.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:22.408 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:22.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:22.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:22.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:22.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:22.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836592 2026-02-20T17:05:22.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836592 2026-02-20T17:05:22.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836592' 2026-02-20T17:05:22.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:22.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673069 2026-02-20T17:05:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673069 2026-02-20T17:05:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836592 1-42949673069' 2026-02-20T17:05:22.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:22.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509547 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509547 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836592 1-42949673069 2-64424509547' 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836592 2026-02-20T17:05:22.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:22.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:22.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836592 2026-02-20T17:05:22.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:22.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836592 2026-02-20T17:05:22.802 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836592 2026-02-20T17:05:22.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2026-02-20T17:05:22.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:22.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836590 -lt 21474836592 2026-02-20T17:05:22.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:23.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:23.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:24.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836593 -lt 21474836592 2026-02-20T17:05:24.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:24.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673069 2026-02-20T17:05:24.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:24.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:24.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673069 2026-02-20T17:05:24.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:24.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673069 2026-02-20T17:05:24.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2026-02-20T17:05:24.125 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673069 2026-02-20T17:05:24.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:24.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673070 -lt 42949673069 2026-02-20T17:05:24.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:24.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509547 2026-02-20T17:05:24.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:24.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:24.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509547 2026-02-20T17:05:24.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:24.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509547 2026-02-20T17:05:24.298 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509547 2026-02-20T17:05:24.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2026-02-20T17:05:24.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:24.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509548 -lt 64424509547 2026-02-20T17:05:24.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:24.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:24.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:24.614 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:24.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:24.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:24.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:24.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:24.624 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:24.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:24.786 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:24.786 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:24.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:24.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:24.786 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836596 2026-02-20T17:05:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836596 2026-02-20T17:05:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596' 2026-02-20T17:05:24.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:24.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:24.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673073 2026-02-20T17:05:24.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673073 2026-02-20T17:05:24.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-42949673073' 2026-02-20T17:05:24.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:24.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509550 2026-02-20T17:05:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509550 2026-02-20T17:05:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-42949673073 2-64424509550' 2026-02-20T17:05:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:25.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836596 2026-02-20T17:05:25.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:25.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:25.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836596 2026-02-20T17:05:25.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:25.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836596 2026-02-20T17:05:25.007 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836596 2026-02-20T17:05:25.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2026-02-20T17:05:25.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:25.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836593 -lt 21474836596 2026-02-20T17:05:25.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:26.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:26.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:26.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836596 -lt 21474836596 2026-02-20T17:05:26.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:26.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673073 2026-02-20T17:05:26.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:26.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:26.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673073 2026-02-20T17:05:26.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:26.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673073 2026-02-20T17:05:26.333 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673073 2026-02-20T17:05:26.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2026-02-20T17:05:26.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:26.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673073 -lt 42949673073 2026-02-20T17:05:26.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:26.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509550 2026-02-20T17:05:26.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:26.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:26.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509550 2026-02-20T17:05:26.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:26.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509550 2026-02-20T17:05:26.494 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509550 2026-02-20T17:05:26.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509550' 2026-02-20T17:05:26.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:26.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509551 -lt 64424509550 2026-02-20T17:05:26.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:26.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:26.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:26.815 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:26.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:26.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:26.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:26.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:26.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:26.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:26.984 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:26.984 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:26.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:26.985 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836599 2026-02-20T17:05:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836599 2026-02-20T17:05:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599' 2026-02-20T17:05:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:27.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673076 2026-02-20T17:05:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673076 2026-02-20T17:05:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-42949673076' 2026-02-20T17:05:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:27.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:27.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509553 2026-02-20T17:05:27.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509553 2026-02-20T17:05:27.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-42949673076 2-64424509553' 2026-02-20T17:05:27.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:27.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836599 2026-02-20T17:05:27.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:27.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:27.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836599 2026-02-20T17:05:27.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:27.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836599 2026-02-20T17:05:27.206 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836599 2026-02-20T17:05:27.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2026-02-20T17:05:27.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:27.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836596 -lt 21474836599 2026-02-20T17:05:27.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:28.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:28.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:28.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836599 -lt 21474836599 2026-02-20T17:05:28.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:28.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673076 2026-02-20T17:05:28.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:28.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:28.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673076 2026-02-20T17:05:28.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:28.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673076 2026-02-20T17:05:28.543 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673076 2026-02-20T17:05:28.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673076' 2026-02-20T17:05:28.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:28.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673076 -lt 42949673076 2026-02-20T17:05:28.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:28.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509553 2026-02-20T17:05:28.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:28.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:28.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509553 2026-02-20T17:05:28.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:28.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509553 2026-02-20T17:05:28.712 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509553 2026-02-20T17:05:28.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509553' 2026-02-20T17:05:28.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:28.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509554 -lt 64424509553 2026-02-20T17:05:28.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:28.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:28.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:29.020 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:29.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:29.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:29.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:29.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:29.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:29.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:29.187 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:29.188 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:29.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:29.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:29.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:29.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836602 2026-02-20T17:05:29.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836602 2026-02-20T17:05:29.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602' 2026-02-20T17:05:29.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:29.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:29.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673079 2026-02-20T17:05:29.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673079 2026-02-20T17:05:29.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602 1-42949673079' 2026-02-20T17:05:29.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:29.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:29.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509557 2026-02-20T17:05:29.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509557 2026-02-20T17:05:29.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602 1-42949673079 2-64424509557' 2026-02-20T17:05:29.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:29.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836602 2026-02-20T17:05:29.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:29.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:29.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836602 2026-02-20T17:05:29.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:29.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836602 2026-02-20T17:05:29.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2026-02-20T17:05:29.409 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836602 2026-02-20T17:05:29.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:29.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836602 -lt 21474836602 2026-02-20T17:05:29.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:29.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673079 2026-02-20T17:05:29.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:29.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:29.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673079 2026-02-20T17:05:29.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:29.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673079 2026-02-20T17:05:29.572 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673079 2026-02-20T17:05:29.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673079' 2026-02-20T17:05:29.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:29.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673079 -lt 42949673079 2026-02-20T17:05:29.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:29.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509557 2026-02-20T17:05:29.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:29.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:29.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509557 2026-02-20T17:05:29.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:29.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509557 2026-02-20T17:05:29.736 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509557 2026-02-20T17:05:29.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509557' 2026-02-20T17:05:29.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:29.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509557 -lt 64424509557 2026-02-20T17:05:29.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:29.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:29.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:30.046 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:30.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:30.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:30.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:30.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:30.056 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:30.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:30.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836604 2026-02-20T17:05:30.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836604 2026-02-20T17:05:30.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604' 2026-02-20T17:05:30.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:30.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:30.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673081 2026-02-20T17:05:30.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673081 2026-02-20T17:05:30.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-42949673081' 2026-02-20T17:05:30.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:30.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:30.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509559 2026-02-20T17:05:30.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509559 2026-02-20T17:05:30.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-42949673081 2-64424509559' 2026-02-20T17:05:30.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:30.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836604 2026-02-20T17:05:30.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:30.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:30.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836604 2026-02-20T17:05:30.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:30.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836604 2026-02-20T17:05:30.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2026-02-20T17:05:30.439 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836604 2026-02-20T17:05:30.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836602 -lt 21474836604 2026-02-20T17:05:30.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:31.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:31.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836605 -lt 21474836604 2026-02-20T17:05:31.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673081 2026-02-20T17:05:31.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:31.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:31.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673081 2026-02-20T17:05:31.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:31.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673081 2026-02-20T17:05:31.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673081' 2026-02-20T17:05:31.760 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673081 2026-02-20T17:05:31.761 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:31.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673082 -lt 42949673081 2026-02-20T17:05:31.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:31.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509559 2026-02-20T17:05:31.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:31.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:31.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509559 2026-02-20T17:05:31.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:31.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509559 2026-02-20T17:05:31.930 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509559 2026-02-20T17:05:31.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509559' 2026-02-20T17:05:31.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:32.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509560 -lt 64424509559 2026-02-20T17:05:32.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:32.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:32.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:32.241 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:32.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:32.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:32.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:32.418 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:32.418 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:32.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:32.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:32.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:32.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836607 2026-02-20T17:05:32.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836607 2026-02-20T17:05:32.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607' 2026-02-20T17:05:32.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:32.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:32.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673084 2026-02-20T17:05:32.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673084 2026-02-20T17:05:32.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-42949673084' 2026-02-20T17:05:32.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:32.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:32.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509562 2026-02-20T17:05:32.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509562 2026-02-20T17:05:32.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509562' 2026-02-20T17:05:32.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:32.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836607 2026-02-20T17:05:32.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:32.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:32.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836607 2026-02-20T17:05:32.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:32.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836607 2026-02-20T17:05:32.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2026-02-20T17:05:32.639 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836607 2026-02-20T17:05:32.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:32.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836605 -lt 21474836607 2026-02-20T17:05:32.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:33.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:33.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836608 -lt 21474836607 2026-02-20T17:05:33.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:33.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673084 2026-02-20T17:05:33.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:33.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:33.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673084 2026-02-20T17:05:33.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:33.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673084 2026-02-20T17:05:33.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673084' 2026-02-20T17:05:33.969 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673084 2026-02-20T17:05:33.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:34.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673085 -lt 42949673084 2026-02-20T17:05:34.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:34.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509562 2026-02-20T17:05:34.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:34.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:34.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509562 2026-02-20T17:05:34.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:34.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509562 2026-02-20T17:05:34.124 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509562 2026-02-20T17:05:34.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509562' 2026-02-20T17:05:34.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:34.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509563 -lt 64424509562 2026-02-20T17:05:34.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:34.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:34.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:34.434 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:34.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:34.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:34.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:34.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:34.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:34.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:34.602 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:34.602 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:34.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:34.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:34.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:34.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836610 2026-02-20T17:05:34.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836610 2026-02-20T17:05:34.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610' 2026-02-20T17:05:34.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:34.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:34.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673087 2026-02-20T17:05:34.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673087 2026-02-20T17:05:34.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-42949673087' 2026-02-20T17:05:34.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:34.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509565 2026-02-20T17:05:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509565 2026-02-20T17:05:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-42949673087 2-64424509565' 2026-02-20T17:05:34.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:34.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836610 2026-02-20T17:05:34.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:34.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:34.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836610 2026-02-20T17:05:34.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:34.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836610 2026-02-20T17:05:34.822 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836610 2026-02-20T17:05:34.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2026-02-20T17:05:34.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:34.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836608 -lt 21474836610 2026-02-20T17:05:34.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:35.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:35.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:36.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836611 -lt 21474836610 2026-02-20T17:05:36.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:36.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673087 2026-02-20T17:05:36.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:36.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:36.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673087 2026-02-20T17:05:36.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:36.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673087 2026-02-20T17:05:36.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673087' 2026-02-20T17:05:36.152 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673087 2026-02-20T17:05:36.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673088 -lt 42949673087 2026-02-20T17:05:36.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:36.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509565 2026-02-20T17:05:36.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:36.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:36.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509565 2026-02-20T17:05:36.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:36.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509565 2026-02-20T17:05:36.316 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509565 2026-02-20T17:05:36.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509565' 2026-02-20T17:05:36.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:36.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509566 -lt 64424509565 2026-02-20T17:05:36.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:36.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:36.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:36.624 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:36.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:36.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:36.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:36.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:36.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:36.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:36.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836614 2026-02-20T17:05:36.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836614 2026-02-20T17:05:36.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614' 2026-02-20T17:05:36.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:36.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:36.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673091 2026-02-20T17:05:36.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673091 2026-02-20T17:05:36.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-42949673091' 2026-02-20T17:05:36.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:36.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:37.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509568 2026-02-20T17:05:37.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509568 2026-02-20T17:05:37.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-42949673091 2-64424509568' 2026-02-20T17:05:37.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:37.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836614 2026-02-20T17:05:37.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:37.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:37.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836614 2026-02-20T17:05:37.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:37.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836614 2026-02-20T17:05:37.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2026-02-20T17:05:37.018 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836614 2026-02-20T17:05:37.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:37.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836611 -lt 21474836614 2026-02-20T17:05:37.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:38.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:38.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:38.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836614 -lt 21474836614 2026-02-20T17:05:38.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:38.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673091 2026-02-20T17:05:38.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:38.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:38.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673091 2026-02-20T17:05:38.345 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:38.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673091 2026-02-20T17:05:38.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673091' 2026-02-20T17:05:38.347 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673091 2026-02-20T17:05:38.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:38.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673091 -lt 42949673091 2026-02-20T17:05:38.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:38.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509568 2026-02-20T17:05:38.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:38.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:38.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509568 2026-02-20T17:05:38.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509568 2026-02-20T17:05:38.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509568' 2026-02-20T17:05:38.522 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509568 2026-02-20T17:05:38.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:38.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509569 -lt 64424509568 2026-02-20T17:05:38.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:38.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:38.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:38.834 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:38.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:38.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:38.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:38.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:38.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:39.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:39.004 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:39.004 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:39.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:39.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:39.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:39.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836617 2026-02-20T17:05:39.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836617 2026-02-20T17:05:39.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617' 2026-02-20T17:05:39.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:39.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:39.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673094 2026-02-20T17:05:39.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673094 2026-02-20T17:05:39.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-42949673094' 2026-02-20T17:05:39.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:39.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:39.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509571 2026-02-20T17:05:39.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509571 2026-02-20T17:05:39.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-42949673094 2-64424509571' 2026-02-20T17:05:39.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:39.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836617 2026-02-20T17:05:39.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:39.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836617 2026-02-20T17:05:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:39.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836617 2026-02-20T17:05:39.226 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836617 2026-02-20T17:05:39.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836617' 2026-02-20T17:05:39.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:39.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836614 -lt 21474836617 2026-02-20T17:05:39.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:40.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:40.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:40.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836617 -lt 21474836617 2026-02-20T17:05:40.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:40.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673094 2026-02-20T17:05:40.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:40.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:40.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673094 2026-02-20T17:05:40.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:40.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673094 2026-02-20T17:05:40.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673094' 2026-02-20T17:05:40.540 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673094 2026-02-20T17:05:40.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:40.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673094 -lt 42949673094 2026-02-20T17:05:40.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:40.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509571 2026-02-20T17:05:40.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:40.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:40.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509571 2026-02-20T17:05:40.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:40.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509571 2026-02-20T17:05:40.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509571' 2026-02-20T17:05:40.704 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509571 2026-02-20T17:05:40.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:40.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509572 -lt 64424509571 2026-02-20T17:05:40.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:40.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:40.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:41.011 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:41.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:41.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:41.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:41.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:41.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:41.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:41.177 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:41.177 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:41.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:41.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:41.178 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:41.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836620 2026-02-20T17:05:41.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836620 2026-02-20T17:05:41.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620' 2026-02-20T17:05:41.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:41.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:41.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673097 2026-02-20T17:05:41.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673097 2026-02-20T17:05:41.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-42949673097' 2026-02-20T17:05:41.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:41.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509575 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509575 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-42949673097 2-64424509575' 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836620 2026-02-20T17:05:41.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:41.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836620 2026-02-20T17:05:41.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:41.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836620 2026-02-20T17:05:41.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836620' 2026-02-20T17:05:41.402 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836620 2026-02-20T17:05:41.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:41.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836620 -lt 21474836620 2026-02-20T17:05:41.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:41.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673097 2026-02-20T17:05:41.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:41.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:41.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673097 2026-02-20T17:05:41.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:41.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673097 2026-02-20T17:05:41.567 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673097 2026-02-20T17:05:41.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673097' 2026-02-20T17:05:41.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:41.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673097 -lt 42949673097 2026-02-20T17:05:41.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:41.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509575 2026-02-20T17:05:41.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:41.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:41.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509575 2026-02-20T17:05:41.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:41.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509575 2026-02-20T17:05:41.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509575' 2026-02-20T17:05:41.733 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509575 2026-02-20T17:05:41.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:41.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509575 -lt 64424509575 2026-02-20T17:05:41.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:41.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:41.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:42.050 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:42.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:42.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:42.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:42.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:42.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:42.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:42.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836622 2026-02-20T17:05:42.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836622 2026-02-20T17:05:42.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622' 2026-02-20T17:05:42.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:42.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:42.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673099 2026-02-20T17:05:42.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673099 2026-02-20T17:05:42.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622 1-42949673099' 2026-02-20T17:05:42.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:42.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:42.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509577 2026-02-20T17:05:42.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509577 2026-02-20T17:05:42.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622 1-42949673099 2-64424509577' 2026-02-20T17:05:42.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:42.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836622 2026-02-20T17:05:42.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:42.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:42.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836622 2026-02-20T17:05:42.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:42.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836622 2026-02-20T17:05:42.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836622' 2026-02-20T17:05:42.431 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836622 2026-02-20T17:05:42.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:42.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836620 -lt 21474836622 2026-02-20T17:05:42.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:43.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:43.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:43.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836623 -lt 21474836622 2026-02-20T17:05:43.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:43.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673099 2026-02-20T17:05:43.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:43.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:43.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673099 2026-02-20T17:05:43.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:43.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673099 2026-02-20T17:05:43.764 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673099 2026-02-20T17:05:43.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673099' 2026-02-20T17:05:43.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:43.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673100 -lt 42949673099 2026-02-20T17:05:43.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:43.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509577 2026-02-20T17:05:43.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:43.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:43.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509577 2026-02-20T17:05:43.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:43.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509577 2026-02-20T17:05:43.930 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509577 2026-02-20T17:05:43.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509577' 2026-02-20T17:05:43.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:44.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509578 -lt 64424509577 2026-02-20T17:05:44.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:44.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:44.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:44.248 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:44.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:44.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:44.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:44.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:44.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:44.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:44.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836625 2026-02-20T17:05:44.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836625 2026-02-20T17:05:44.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625' 2026-02-20T17:05:44.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:44.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:44.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673102 2026-02-20T17:05:44.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673102 2026-02-20T17:05:44.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625 1-42949673102' 2026-02-20T17:05:44.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:44.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:44.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509580 2026-02-20T17:05:44.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509580 2026-02-20T17:05:44.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625 1-42949673102 2-64424509580' 2026-02-20T17:05:44.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:44.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836625 2026-02-20T17:05:44.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:44.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:44.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836625 2026-02-20T17:05:44.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:44.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836625 2026-02-20T17:05:44.633 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836625 2026-02-20T17:05:44.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836625' 2026-02-20T17:05:44.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:44.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836623 -lt 21474836625 2026-02-20T17:05:44.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:45.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:45.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:45.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836626 -lt 21474836625 2026-02-20T17:05:45.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:45.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673102 2026-02-20T17:05:45.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:45.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:45.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673102 2026-02-20T17:05:45.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:45.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673102 2026-02-20T17:05:45.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673102' 2026-02-20T17:05:45.953 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673102 2026-02-20T17:05:45.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:46.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673103 -lt 42949673102 2026-02-20T17:05:46.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:46.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509580 2026-02-20T17:05:46.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:46.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:46.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509580 2026-02-20T17:05:46.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:46.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509580 2026-02-20T17:05:46.118 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509580 2026-02-20T17:05:46.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509580' 2026-02-20T17:05:46.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:46.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509581 -lt 64424509580 2026-02-20T17:05:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:46.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:46.423 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:46.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:46.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:46.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:46.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:46.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:46.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:46.592 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:46.592 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:46.592 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:46.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:46.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:46.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836628 2026-02-20T17:05:46.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836628 2026-02-20T17:05:46.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628' 2026-02-20T17:05:46.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:46.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:46.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673105 2026-02-20T17:05:46.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673105 2026-02-20T17:05:46.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628 1-42949673105' 2026-02-20T17:05:46.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:46.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509583 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509583 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628 1-42949673105 2-64424509583' 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836628 2026-02-20T17:05:46.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:46.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:46.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836628 2026-02-20T17:05:46.813 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:46.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836628 2026-02-20T17:05:46.815 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836628 2026-02-20T17:05:46.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836628' 2026-02-20T17:05:46.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:46.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836626 -lt 21474836628 2026-02-20T17:05:46.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:47.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:47.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:48.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836628 2026-02-20T17:05:48.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:48.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673105 2026-02-20T17:05:48.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:48.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:48.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673105 2026-02-20T17:05:48.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:48.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673105 2026-02-20T17:05:48.135 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673105 2026-02-20T17:05:48.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673105' 2026-02-20T17:05:48.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:48.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673106 -lt 42949673105 2026-02-20T17:05:48.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:48.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509583 2026-02-20T17:05:48.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:48.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:48.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509583 2026-02-20T17:05:48.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:48.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509583 2026-02-20T17:05:48.296 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509583 2026-02-20T17:05:48.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509583' 2026-02-20T17:05:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:48.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509584 -lt 64424509583 2026-02-20T17:05:48.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:48.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:48.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:48.601 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:48.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:48.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:48.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:48.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:48.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:48.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:48.771 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:48.771 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:48.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:48.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:48.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836632 2026-02-20T17:05:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836632 2026-02-20T17:05:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632' 2026-02-20T17:05:48.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:48.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:48.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673109 2026-02-20T17:05:48.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673109 2026-02-20T17:05:48.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-42949673109' 2026-02-20T17:05:48.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:48.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:48.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509586 2026-02-20T17:05:48.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509586 2026-02-20T17:05:48.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-42949673109 2-64424509586' 2026-02-20T17:05:48.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:48.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836632 2026-02-20T17:05:48.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:48.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:48.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836632 2026-02-20T17:05:48.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:48.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836632 2026-02-20T17:05:48.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836632' 2026-02-20T17:05:48.990 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836632 2026-02-20T17:05:48.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:49.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836632 2026-02-20T17:05:49.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:50.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:50.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:50.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836632 2026-02-20T17:05:50.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:50.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673109 2026-02-20T17:05:50.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:50.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:50.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673109 2026-02-20T17:05:50.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:50.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673109 2026-02-20T17:05:50.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673109' 2026-02-20T17:05:50.317 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673109 2026-02-20T17:05:50.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:50.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673109 -lt 42949673109 2026-02-20T17:05:50.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:50.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509586 2026-02-20T17:05:50.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:50.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509586 2026-02-20T17:05:50.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:50.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509586 2026-02-20T17:05:50.473 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509586 2026-02-20T17:05:50.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509586' 2026-02-20T17:05:50.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:50.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509587 -lt 64424509586 2026-02-20T17:05:50.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:50.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:50.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:50.783 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:50.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:50.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:50.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:50.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:50.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:50.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:51.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836635 2026-02-20T17:05:51.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836635 2026-02-20T17:05:51.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635' 2026-02-20T17:05:51.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:51.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:51.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673112 2026-02-20T17:05:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673112 2026-02-20T17:05:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-42949673112' 2026-02-20T17:05:51.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:51.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:51.170 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509589 2026-02-20T17:05:51.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509589 2026-02-20T17:05:51.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-42949673112 2-64424509589' 2026-02-20T17:05:51.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:51.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836635 2026-02-20T17:05:51.171 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:51.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:51.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836635 2026-02-20T17:05:51.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:51.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836635 2026-02-20T17:05:51.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836635' 2026-02-20T17:05:51.175 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836635 2026-02-20T17:05:51.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:51.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836635 2026-02-20T17:05:51.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:52.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:52.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:52.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836635 2026-02-20T17:05:52.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:52.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673112 2026-02-20T17:05:52.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:52.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:52.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673112 2026-02-20T17:05:52.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:52.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673112 2026-02-20T17:05:52.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673112' 2026-02-20T17:05:52.507 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673112 2026-02-20T17:05:52.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:52.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673112 -lt 42949673112 2026-02-20T17:05:52.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:52.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509589 2026-02-20T17:05:52.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:52.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:52.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509589 2026-02-20T17:05:52.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:52.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509589 2026-02-20T17:05:52.679 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509589 2026-02-20T17:05:52.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509589' 2026-02-20T17:05:52.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:52.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509590 -lt 64424509589 2026-02-20T17:05:52.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:52.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:52.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:52.987 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:52.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:52.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:52.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:52.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:52.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:53.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:53.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836638 2026-02-20T17:05:53.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836638 2026-02-20T17:05:53.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638' 2026-02-20T17:05:53.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:53.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:53.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673115 2026-02-20T17:05:53.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673115 2026-02-20T17:05:53.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-42949673115' 2026-02-20T17:05:53.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:53.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509592 2026-02-20T17:05:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509592 2026-02-20T17:05:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-42949673115 2-64424509592' 2026-02-20T17:05:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:53.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836638 2026-02-20T17:05:53.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:53.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:53.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836638 2026-02-20T17:05:53.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:53.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836638 2026-02-20T17:05:53.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836638' 2026-02-20T17:05:53.370 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836638 2026-02-20T17:05:53.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:53.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836638 2026-02-20T17:05:53.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673115 2026-02-20T17:05:53.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:53.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:53.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673115 2026-02-20T17:05:53.536 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:53.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673115 2026-02-20T17:05:53.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673115' 2026-02-20T17:05:53.537 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673115 2026-02-20T17:05:53.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:53.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673115 -lt 42949673115 2026-02-20T17:05:53.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:53.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509592 2026-02-20T17:05:53.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:53.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:53.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509592 2026-02-20T17:05:53.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:53.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509592 2026-02-20T17:05:53.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509592' 2026-02-20T17:05:53.700 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509592 2026-02-20T17:05:53.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:53.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509593 -lt 64424509592 2026-02-20T17:05:53.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:53.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:53.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:54.012 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:54.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:54.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:54.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:54.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:54.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:54.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:54.176 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:54.176 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:54.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:54.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:54.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:54.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836640 2026-02-20T17:05:54.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836640 2026-02-20T17:05:54.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640' 2026-02-20T17:05:54.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:54.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:54.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673117 2026-02-20T17:05:54.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673117 2026-02-20T17:05:54.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-42949673117' 2026-02-20T17:05:54.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:54.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:54.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509595 2026-02-20T17:05:54.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509595 2026-02-20T17:05:54.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-42949673117 2-64424509595' 2026-02-20T17:05:54.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:54.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836640 2026-02-20T17:05:54.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:54.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:54.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836640 2026-02-20T17:05:54.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836640 2026-02-20T17:05:54.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836640' 2026-02-20T17:05:54.398 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836640 2026-02-20T17:05:54.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:54.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836640 2026-02-20T17:05:54.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:55.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:55.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:55.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836640 2026-02-20T17:05:55.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:55.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673117 2026-02-20T17:05:55.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:55.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:55.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673117 2026-02-20T17:05:55.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:55.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673117 2026-02-20T17:05:55.718 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673117 2026-02-20T17:05:55.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673117' 2026-02-20T17:05:55.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:55.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673118 -lt 42949673117 2026-02-20T17:05:55.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:55.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509595 2026-02-20T17:05:55.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:55.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:55.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509595 2026-02-20T17:05:55.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:55.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509595 2026-02-20T17:05:55.885 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509595 2026-02-20T17:05:55.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509595' 2026-02-20T17:05:55.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:56.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509596 -lt 64424509595 2026-02-20T17:05:56.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:56.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:56.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:56.204 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:56.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:56.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:56.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:56.214 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:56.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:56.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:56.367 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:56.368 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:56.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:56.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:56.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:56.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836643 2026-02-20T17:05:56.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836643 2026-02-20T17:05:56.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643' 2026-02-20T17:05:56.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:56.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:56.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673120 2026-02-20T17:05:56.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673120 2026-02-20T17:05:56.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-42949673120' 2026-02-20T17:05:56.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:56.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:56.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509598 2026-02-20T17:05:56.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509598 2026-02-20T17:05:56.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-42949673120 2-64424509598' 2026-02-20T17:05:56.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:56.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836643 2026-02-20T17:05:56.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:56.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:56.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836643 2026-02-20T17:05:56.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:56.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836643 2026-02-20T17:05:56.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836643' 2026-02-20T17:05:56.586 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836643 2026-02-20T17:05:56.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:56.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836643 2026-02-20T17:05:56.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:57.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:57.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:57.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836643 2026-02-20T17:05:57.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:57.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673120 2026-02-20T17:05:57.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:57.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:05:57.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673120 2026-02-20T17:05:57.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:57.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673120 2026-02-20T17:05:57.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673120' 2026-02-20T17:05:57.915 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673120 2026-02-20T17:05:57.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:05:58.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673121 -lt 42949673120 2026-02-20T17:05:58.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:58.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509598 2026-02-20T17:05:58.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:58.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:05:58.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509598 2026-02-20T17:05:58.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:58.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509598 2026-02-20T17:05:58.095 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509598 2026-02-20T17:05:58.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509598' 2026-02-20T17:05:58.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:05:58.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509599 -lt 64424509598 2026-02-20T17:05:58.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:05:58.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:05:58.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:05:58.402 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:05:58.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:05:58.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:05:58.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:05:58.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:05:58.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:05:58.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:05:58.577 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:05:58.577 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:05:58.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:05:58.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:58.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:05:58.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836646 2026-02-20T17:05:58.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836646 2026-02-20T17:05:58.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646' 2026-02-20T17:05:58.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:58.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:05:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673123 2026-02-20T17:05:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673123 2026-02-20T17:05:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-42949673123' 2026-02-20T17:05:58.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:05:58.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:05:58.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509601 2026-02-20T17:05:58.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509601 2026-02-20T17:05:58.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-42949673123 2-64424509601' 2026-02-20T17:05:58.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:05:58.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836646 2026-02-20T17:05:58.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:05:58.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:05:58.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836646 2026-02-20T17:05:58.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:05:58.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836646 2026-02-20T17:05:58.805 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836646 2026-02-20T17:05:58.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836646' 2026-02-20T17:05:58.806 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:05:58.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836646 2026-02-20T17:05:58.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:05:59.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:05:59.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:00.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836646 2026-02-20T17:06:00.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:00.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673123 2026-02-20T17:06:00.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:00.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:00.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673123 2026-02-20T17:06:00.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:00.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673123 2026-02-20T17:06:00.127 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673123 2026-02-20T17:06:00.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673123' 2026-02-20T17:06:00.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:00.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673124 -lt 42949673123 2026-02-20T17:06:00.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:00.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509601 2026-02-20T17:06:00.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:00.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:00.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509601 2026-02-20T17:06:00.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:00.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509601 2026-02-20T17:06:00.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509601' 2026-02-20T17:06:00.301 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509601 2026-02-20T17:06:00.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:00.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509602 -lt 64424509601 2026-02-20T17:06:00.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:00.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:00.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:00.609 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:00.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:00.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:00.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:00.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:00.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:00.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:00.778 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:00.778 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:00.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:00.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:00.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:00.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836650 2026-02-20T17:06:00.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836650 2026-02-20T17:06:00.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650' 2026-02-20T17:06:00.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:00.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:00.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673127 2026-02-20T17:06:00.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673127 2026-02-20T17:06:00.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-42949673127' 2026-02-20T17:06:00.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:00.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:00.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509604 2026-02-20T17:06:00.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509604 2026-02-20T17:06:00.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-42949673127 2-64424509604' 2026-02-20T17:06:00.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:00.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836650 2026-02-20T17:06:00.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:00.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:00.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836650 2026-02-20T17:06:00.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:00.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836650 2026-02-20T17:06:00.998 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836650 2026-02-20T17:06:00.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836650' 2026-02-20T17:06:00.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836650 2026-02-20T17:06:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:02.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:02.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:02.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836650 2026-02-20T17:06:02.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:02.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673127 2026-02-20T17:06:02.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:02.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:02.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673127 2026-02-20T17:06:02.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:02.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673127 2026-02-20T17:06:02.325 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673127 2026-02-20T17:06:02.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673127' 2026-02-20T17:06:02.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:02.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673127 -lt 42949673127 2026-02-20T17:06:02.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:02.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509604 2026-02-20T17:06:02.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:02.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:02.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509604 2026-02-20T17:06:02.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:02.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509604 2026-02-20T17:06:02.491 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509604 2026-02-20T17:06:02.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509604' 2026-02-20T17:06:02.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:02.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509605 -lt 64424509604 2026-02-20T17:06:02.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:02.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:02.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:02.806 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:02.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:02.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:02.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:02.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:02.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:02.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:02.977 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:02.977 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:02.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:02.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:02.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:03.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836653 2026-02-20T17:06:03.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836653 2026-02-20T17:06:03.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653' 2026-02-20T17:06:03.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:03.053 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673130 2026-02-20T17:06:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673130 2026-02-20T17:06:03.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-42949673130' 2026-02-20T17:06:03.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:03.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:03.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509607 2026-02-20T17:06:03.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509607 2026-02-20T17:06:03.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-42949673130 2-64424509607' 2026-02-20T17:06:03.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:03.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836653 2026-02-20T17:06:03.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:03.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:03.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836653 2026-02-20T17:06:03.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:03.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836653 2026-02-20T17:06:03.199 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836653 2026-02-20T17:06:03.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836653' 2026-02-20T17:06:03.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:03.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836653 2026-02-20T17:06:03.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:04.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:04.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:04.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836653 2026-02-20T17:06:04.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:04.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673130 2026-02-20T17:06:04.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:04.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:04.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673130 2026-02-20T17:06:04.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:04.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673130 2026-02-20T17:06:04.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673130' 2026-02-20T17:06:04.529 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673130 2026-02-20T17:06:04.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:04.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673130 -lt 42949673130 2026-02-20T17:06:04.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:04.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509607 2026-02-20T17:06:04.694 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:04.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:04.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509607 2026-02-20T17:06:04.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:04.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509607 2026-02-20T17:06:04.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509607' 2026-02-20T17:06:04.698 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509607 2026-02-20T17:06:04.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:04.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509608 -lt 64424509607 2026-02-20T17:06:04.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:04.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:04.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:05.017 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:05.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:05.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:05.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:05.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:05.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:05.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:05.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836656 2026-02-20T17:06:05.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836656 2026-02-20T17:06:05.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656' 2026-02-20T17:06:05.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:05.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:05.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673133 2026-02-20T17:06:05.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673133 2026-02-20T17:06:05.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-42949673133' 2026-02-20T17:06:05.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:05.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:05.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509611 2026-02-20T17:06:05.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509611 2026-02-20T17:06:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-42949673133 2-64424509611' 2026-02-20T17:06:05.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:05.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836656 2026-02-20T17:06:05.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:05.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:05.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836656 2026-02-20T17:06:05.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:05.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836656 2026-02-20T17:06:05.411 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836656 2026-02-20T17:06:05.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836656' 2026-02-20T17:06:05.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836656 2026-02-20T17:06:05.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:05.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673133 2026-02-20T17:06:05.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:05.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:05.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673133 2026-02-20T17:06:05.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:05.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673133 2026-02-20T17:06:05.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673133' 2026-02-20T17:06:05.583 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673133 2026-02-20T17:06:05.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:05.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673133 -lt 42949673133 2026-02-20T17:06:05.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:05.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509611 2026-02-20T17:06:05.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:05.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:05.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509611 2026-02-20T17:06:05.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:05.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509611 2026-02-20T17:06:05.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509611' 2026-02-20T17:06:05.753 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509611 2026-02-20T17:06:05.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:05.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509611 -lt 64424509611 2026-02-20T17:06:05.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:05.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:05.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:06.068 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:06.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:06.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:06.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:06.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:06.080 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:06.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:06.235 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:06.235 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:06.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:06.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:06.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:06.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-20T17:06:06.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-20T17:06:06.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-20T17:06:06.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:06.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:06.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673135 2026-02-20T17:06:06.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673135 2026-02-20T17:06:06.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-42949673135' 2026-02-20T17:06:06.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:06.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:06.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509613 2026-02-20T17:06:06.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509613 2026-02-20T17:06:06.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-42949673135 2-64424509613' 2026-02-20T17:06:06.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:06.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-20T17:06:06.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:06.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:06.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-20T17:06:06.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:06.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-20T17:06:06.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-20T17:06:06.458 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836658 2026-02-20T17:06:06.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:06.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836658 2026-02-20T17:06:06.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:07.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:07.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:07.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836658 2026-02-20T17:06:07.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:07.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673135 2026-02-20T17:06:07.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:07.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673135 2026-02-20T17:06:07.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:07.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673135 2026-02-20T17:06:07.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673135' 2026-02-20T17:06:07.783 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673135 2026-02-20T17:06:07.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673136 -lt 42949673135 2026-02-20T17:06:07.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:07.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509613 2026-02-20T17:06:07.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:07.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:07.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509613 2026-02-20T17:06:07.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:07.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509613 2026-02-20T17:06:07.958 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509613 2026-02-20T17:06:07.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509613' 2026-02-20T17:06:07.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:08.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509614 -lt 64424509613 2026-02-20T17:06:08.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:08.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:08.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:08.273 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:08.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:08.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:08.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:08.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:08.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:08.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:08.442 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:08.442 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:08.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:08.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:08.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:08.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-20T17:06:08.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-20T17:06:08.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-20T17:06:08.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:08.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:08.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673138 2026-02-20T17:06:08.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673138 2026-02-20T17:06:08.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-42949673138' 2026-02-20T17:06:08.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:08.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:08.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509616 2026-02-20T17:06:08.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509616 2026-02-20T17:06:08.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-42949673138 2-64424509616' 2026-02-20T17:06:08.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:08.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-20T17:06:08.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:08.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:08.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-20T17:06:08.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:08.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-20T17:06:08.668 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836661 2026-02-20T17:06:08.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-20T17:06:08.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:08.831 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836661 2026-02-20T17:06:08.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:09.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:09.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:09.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836661 2026-02-20T17:06:09.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:09.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673138 2026-02-20T17:06:09.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:09.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:09.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673138 2026-02-20T17:06:09.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:09.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673138 2026-02-20T17:06:09.998 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673138 2026-02-20T17:06:09.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673138' 2026-02-20T17:06:09.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:10.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673139 -lt 42949673138 2026-02-20T17:06:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:10.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509616 2026-02-20T17:06:10.161 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:10.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:10.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509616 2026-02-20T17:06:10.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:10.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509616 2026-02-20T17:06:10.165 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509616 2026-02-20T17:06:10.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509616' 2026-02-20T17:06:10.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:10.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509617 -lt 64424509616 2026-02-20T17:06:10.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:10.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:10.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:10.479 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:10.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:10.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:10.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:10.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:10.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:10.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:10.642 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:10.642 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:10.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:10.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:10.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:10.716 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-20T17:06:10.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-20T17:06:10.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-20T17:06:10.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:10.717 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673141 2026-02-20T17:06:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673141 2026-02-20T17:06:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-42949673141' 2026-02-20T17:06:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:10.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:10.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509619 2026-02-20T17:06:10.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509619 2026-02-20T17:06:10.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-42949673141 2-64424509619' 2026-02-20T17:06:10.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:10.862 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-20T17:06:10.862 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:10.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:10.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-20T17:06:10.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:10.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-20T17:06:10.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-20T17:06:10.865 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836664 2026-02-20T17:06:10.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:11.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836664 2026-02-20T17:06:11.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:12.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:12.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:12.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836664 2026-02-20T17:06:12.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:12.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673141 2026-02-20T17:06:12.192 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:12.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:12.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673141 2026-02-20T17:06:12.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:12.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673141 2026-02-20T17:06:12.196 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673141 2026-02-20T17:06:12.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673141' 2026-02-20T17:06:12.197 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:12.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673142 -lt 42949673141 2026-02-20T17:06:12.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:12.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509619 2026-02-20T17:06:12.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:12.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:12.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509619 2026-02-20T17:06:12.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:12.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509619 2026-02-20T17:06:12.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509619' 2026-02-20T17:06:12.360 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509619 2026-02-20T17:06:12.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:12.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509620 -lt 64424509619 2026-02-20T17:06:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:12.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:12.672 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:12.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:12.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:12.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:12.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:12.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:12.843 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:12.843 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:12.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:12.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:12.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836668 2026-02-20T17:06:12.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836668 2026-02-20T17:06:12.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668' 2026-02-20T17:06:12.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:12.915 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:12.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673145 2026-02-20T17:06:12.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673145 2026-02-20T17:06:12.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-42949673145' 2026-02-20T17:06:12.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:12.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:13.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509622 2026-02-20T17:06:13.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509622 2026-02-20T17:06:13.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-42949673145 2-64424509622' 2026-02-20T17:06:13.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:13.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836668 2026-02-20T17:06:13.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:13.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:13.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836668 2026-02-20T17:06:13.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:13.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836668 2026-02-20T17:06:13.061 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836668 2026-02-20T17:06:13.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836668' 2026-02-20T17:06:13.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:13.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836668 2026-02-20T17:06:13.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:14.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:14.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:14.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836668 2026-02-20T17:06:14.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:14.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673145 2026-02-20T17:06:14.388 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:14.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:14.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673145 2026-02-20T17:06:14.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673145 2026-02-20T17:06:14.391 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673145 2026-02-20T17:06:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673145' 2026-02-20T17:06:14.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:14.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673145 -lt 42949673145 2026-02-20T17:06:14.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:14.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509622 2026-02-20T17:06:14.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:14.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509622 2026-02-20T17:06:14.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:14.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509622 2026-02-20T17:06:14.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509622' 2026-02-20T17:06:14.560 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509622 2026-02-20T17:06:14.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:14.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509623 -lt 64424509622 2026-02-20T17:06:14.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:14.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:14.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:14.874 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:14.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:14.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:14.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:14.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:14.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:15.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:15.037 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:15.037 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:15.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:15.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:15.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:15.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836671 2026-02-20T17:06:15.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836671 2026-02-20T17:06:15.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671' 2026-02-20T17:06:15.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:15.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:15.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673148 2026-02-20T17:06:15.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673148 2026-02-20T17:06:15.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-42949673148' 2026-02-20T17:06:15.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:15.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509625 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509625 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-42949673148 2-64424509625' 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836671 2026-02-20T17:06:15.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:15.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:15.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836671 2026-02-20T17:06:15.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:15.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836671 2026-02-20T17:06:15.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836671' 2026-02-20T17:06:15.258 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836671 2026-02-20T17:06:15.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:15.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836671 2026-02-20T17:06:15.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:16.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:16.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:16.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836671 -lt 21474836671 2026-02-20T17:06:16.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:16.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673148 2026-02-20T17:06:16.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:16.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:16.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673148 2026-02-20T17:06:16.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:16.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673148 2026-02-20T17:06:16.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673148' 2026-02-20T17:06:16.593 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673148 2026-02-20T17:06:16.593 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:16.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673148 -lt 42949673148 2026-02-20T17:06:16.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:16.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509625 2026-02-20T17:06:16.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:16.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:16.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509625 2026-02-20T17:06:16.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:16.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509625 2026-02-20T17:06:16.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509625' 2026-02-20T17:06:16.757 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509625 2026-02-20T17:06:16.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:16.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509626 -lt 64424509625 2026-02-20T17:06:16.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:16.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:16.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:17.062 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:17.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:17.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:17.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:17.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:17.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:17.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:17.228 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:17.228 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:17.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:17.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:17.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836674 2026-02-20T17:06:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836674 2026-02-20T17:06:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674' 2026-02-20T17:06:17.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:17.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:17.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673151 2026-02-20T17:06:17.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673151 2026-02-20T17:06:17.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-42949673151' 2026-02-20T17:06:17.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:17.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:17.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509629 2026-02-20T17:06:17.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509629 2026-02-20T17:06:17.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-42949673151 2-64424509629' 2026-02-20T17:06:17.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:17.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836674 2026-02-20T17:06:17.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:17.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:17.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836674 2026-02-20T17:06:17.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:17.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836674 2026-02-20T17:06:17.446 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836674 2026-02-20T17:06:17.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836674' 2026-02-20T17:06:17.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:17.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836674 -lt 21474836674 2026-02-20T17:06:17.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:17.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673151 2026-02-20T17:06:17.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:17.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:17.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673151 2026-02-20T17:06:17.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:17.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673151 2026-02-20T17:06:17.604 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673151 2026-02-20T17:06:17.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673151' 2026-02-20T17:06:17.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:17.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673151 -lt 42949673151 2026-02-20T17:06:17.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:17.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509629 2026-02-20T17:06:17.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:17.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:17.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509629 2026-02-20T17:06:17.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:17.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509629 2026-02-20T17:06:17.769 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509629 2026-02-20T17:06:17.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509629' 2026-02-20T17:06:17.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:17.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509628 -lt 64424509629 2026-02-20T17:06:17.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:18.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:18.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:19.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509628 -lt 64424509629 2026-02-20T17:06:19.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:20.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:06:20.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:20.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509631 -lt 64424509629 2026-02-20T17:06:20.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:20.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:20.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:20.396 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:20.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:20.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:20.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:20.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:20.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:20.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:20.560 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:20.560 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:20.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:20.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:20.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:20.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836678 2026-02-20T17:06:20.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836678 2026-02-20T17:06:20.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678' 2026-02-20T17:06:20.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:20.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:20.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673155 2026-02-20T17:06:20.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673155 2026-02-20T17:06:20.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-42949673155' 2026-02-20T17:06:20.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:20.707 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:20.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509633 2026-02-20T17:06:20.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509633 2026-02-20T17:06:20.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-42949673155 2-64424509633' 2026-02-20T17:06:20.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:20.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836678 2026-02-20T17:06:20.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:20.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:20.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836678 2026-02-20T17:06:20.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:20.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836678 2026-02-20T17:06:20.782 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836678 2026-02-20T17:06:20.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836678' 2026-02-20T17:06:20.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:20.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836676 -lt 21474836678 2026-02-20T17:06:20.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:21.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:21.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836678 2026-02-20T17:06:22.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673155 2026-02-20T17:06:22.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:22.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:22.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673155 2026-02-20T17:06:22.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:22.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673155 2026-02-20T17:06:22.114 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673155 2026-02-20T17:06:22.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673155' 2026-02-20T17:06:22.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:22.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673156 -lt 42949673155 2026-02-20T17:06:22.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:22.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509633 2026-02-20T17:06:22.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:22.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:22.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509633 2026-02-20T17:06:22.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:22.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509633 2026-02-20T17:06:22.281 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509633 2026-02-20T17:06:22.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509633' 2026-02-20T17:06:22.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:22.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509634 -lt 64424509633 2026-02-20T17:06:22.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:22.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:22.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:22.589 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:22.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:22.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:22.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:22.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:22.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:22.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:22.756 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:22.756 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:22.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:22.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:22.757 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:22.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836682 2026-02-20T17:06:22.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836682 2026-02-20T17:06:22.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682' 2026-02-20T17:06:22.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:22.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:22.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673159 2026-02-20T17:06:22.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673159 2026-02-20T17:06:22.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-42949673159' 2026-02-20T17:06:22.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:22.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509636 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509636 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-42949673159 2-64424509636' 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836682 2026-02-20T17:06:22.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:22.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:22.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836682 2026-02-20T17:06:22.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:22.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836682 2026-02-20T17:06:22.978 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836682 2026-02-20T17:06:22.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836682' 2026-02-20T17:06:22.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:23.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836682 2026-02-20T17:06:23.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:24.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:24.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:24.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836682 2026-02-20T17:06:24.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:24.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673159 2026-02-20T17:06:24.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:24.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:24.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673159 2026-02-20T17:06:24.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:24.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673159 2026-02-20T17:06:24.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673159' 2026-02-20T17:06:24.316 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673159 2026-02-20T17:06:24.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:24.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673159 -lt 42949673159 2026-02-20T17:06:24.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:24.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509636 2026-02-20T17:06:24.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:24.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:24.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509636 2026-02-20T17:06:24.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:24.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509636 2026-02-20T17:06:24.479 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509636 2026-02-20T17:06:24.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509636' 2026-02-20T17:06:24.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:24.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509637 -lt 64424509636 2026-02-20T17:06:24.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:24.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:24.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:24.793 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:24.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:24.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:24.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:24.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:24.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:24.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:25.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836685 2026-02-20T17:06:25.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836685 2026-02-20T17:06:25.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685' 2026-02-20T17:06:25.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:25.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:25.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673162 2026-02-20T17:06:25.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673162 2026-02-20T17:06:25.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-42949673162' 2026-02-20T17:06:25.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:25.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509639 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509639 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-42949673162 2-64424509639' 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836685 2026-02-20T17:06:25.185 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:25.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:25.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836685 2026-02-20T17:06:25.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:25.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836685 2026-02-20T17:06:25.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836685' 2026-02-20T17:06:25.190 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836685 2026-02-20T17:06:25.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:25.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836685 2026-02-20T17:06:25.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:26.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:26.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:26.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836685 -lt 21474836685 2026-02-20T17:06:26.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:26.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673162 2026-02-20T17:06:26.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:26.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:26.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673162 2026-02-20T17:06:26.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:26.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673162 2026-02-20T17:06:26.511 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673162 2026-02-20T17:06:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673162' 2026-02-20T17:06:26.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:26.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673162 -lt 42949673162 2026-02-20T17:06:26.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:26.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509639 2026-02-20T17:06:26.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:26.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:26.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509639 2026-02-20T17:06:26.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:26.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509639 2026-02-20T17:06:26.680 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509639 2026-02-20T17:06:26.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509639' 2026-02-20T17:06:26.681 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:26.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509640 -lt 64424509639 2026-02-20T17:06:26.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:26.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:26.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:26.984 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:26.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:26.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:26.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:26.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:26.994 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:27.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836688 2026-02-20T17:06:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836688 2026-02-20T17:06:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688' 2026-02-20T17:06:27.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:27.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:27.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673165 2026-02-20T17:06:27.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673165 2026-02-20T17:06:27.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-42949673165' 2026-02-20T17:06:27.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:27.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:27.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509642 2026-02-20T17:06:27.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509642 2026-02-20T17:06:27.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-42949673165 2-64424509642' 2026-02-20T17:06:27.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:27.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836688 2026-02-20T17:06:27.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:27.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:27.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836688 2026-02-20T17:06:27.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:27.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836688 2026-02-20T17:06:27.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836688' 2026-02-20T17:06:27.369 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836688 2026-02-20T17:06:27.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:27.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836688 2026-02-20T17:06:27.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:27.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673165 2026-02-20T17:06:27.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:27.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:27.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673165 2026-02-20T17:06:27.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:27.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673165 2026-02-20T17:06:27.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673165' 2026-02-20T17:06:27.532 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673165 2026-02-20T17:06:27.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673165 -lt 42949673165 2026-02-20T17:06:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:27.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509642 2026-02-20T17:06:27.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:27.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509642 2026-02-20T17:06:27.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:27.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509642 2026-02-20T17:06:27.689 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509642 2026-02-20T17:06:27.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509642' 2026-02-20T17:06:27.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:27.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509643 -lt 64424509642 2026-02-20T17:06:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:27.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:28.004 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:28.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:28.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:28.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:28.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:28.014 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:28.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:28.175 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:28.175 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:28.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:28.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:28.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:28.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836690 2026-02-20T17:06:28.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836690 2026-02-20T17:06:28.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690' 2026-02-20T17:06:28.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:28.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:28.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673167 2026-02-20T17:06:28.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673167 2026-02-20T17:06:28.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-42949673167' 2026-02-20T17:06:28.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:28.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:28.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509645 2026-02-20T17:06:28.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509645 2026-02-20T17:06:28.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-42949673167 2-64424509645' 2026-02-20T17:06:28.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:28.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836690 2026-02-20T17:06:28.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:28.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:28.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836690 2026-02-20T17:06:28.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:28.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836690 2026-02-20T17:06:28.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836690' 2026-02-20T17:06:28.394 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836690 2026-02-20T17:06:28.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:28.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836690 2026-02-20T17:06:28.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:29.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:29.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:29.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836690 2026-02-20T17:06:29.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:29.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673167 2026-02-20T17:06:29.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:29.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:29.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673167 2026-02-20T17:06:29.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:29.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673167 2026-02-20T17:06:29.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673167' 2026-02-20T17:06:29.729 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673167 2026-02-20T17:06:29.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:29.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673168 -lt 42949673167 2026-02-20T17:06:29.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:29.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509645 2026-02-20T17:06:29.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:29.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:29.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509645 2026-02-20T17:06:29.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:29.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509645 2026-02-20T17:06:29.891 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509645 2026-02-20T17:06:29.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509645' 2026-02-20T17:06:29.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:30.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509646 -lt 64424509645 2026-02-20T17:06:30.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:30.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:30.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:30.207 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:30.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:30.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:30.379 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:30.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836693 2026-02-20T17:06:30.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836693 2026-02-20T17:06:30.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693' 2026-02-20T17:06:30.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:30.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:30.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673170 2026-02-20T17:06:30.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673170 2026-02-20T17:06:30.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-42949673170' 2026-02-20T17:06:30.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:30.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:30.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509648 2026-02-20T17:06:30.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509648 2026-02-20T17:06:30.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-42949673170 2-64424509648' 2026-02-20T17:06:30.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:30.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836693 2026-02-20T17:06:30.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:30.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:30.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836693 2026-02-20T17:06:30.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836693 2026-02-20T17:06:30.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836693' 2026-02-20T17:06:30.599 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836693 2026-02-20T17:06:30.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:30.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836693 2026-02-20T17:06:30.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:31.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:31.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836694 -lt 21474836693 2026-02-20T17:06:31.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:31.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673170 2026-02-20T17:06:31.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:31.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:31.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673170 2026-02-20T17:06:31.925 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:31.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673170 2026-02-20T17:06:31.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673170' 2026-02-20T17:06:31.927 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673170 2026-02-20T17:06:31.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:32.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673171 -lt 42949673170 2026-02-20T17:06:32.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:32.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509648 2026-02-20T17:06:32.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:32.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509648 2026-02-20T17:06:32.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509648 2026-02-20T17:06:32.101 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509648 2026-02-20T17:06:32.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509648' 2026-02-20T17:06:32.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:32.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509649 -lt 64424509648 2026-02-20T17:06:32.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:32.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:32.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:32.408 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:32.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:06:32.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:06:32.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:06:32.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:06:32.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:06:32.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:06:32.580 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:06:32.580 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:06:32.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:06:32.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:06:32.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836696 2026-02-20T17:06:32.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836696 2026-02-20T17:06:32.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696' 2026-02-20T17:06:32.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:32.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:06:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673173 2026-02-20T17:06:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673173 2026-02-20T17:06:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696 1-42949673173' 2026-02-20T17:06:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:06:32.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:06:32.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509651 2026-02-20T17:06:32.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509651 2026-02-20T17:06:32.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696 1-42949673173 2-64424509651' 2026-02-20T17:06:32.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:32.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836696 2026-02-20T17:06:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:32.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:06:32.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836696 2026-02-20T17:06:32.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:32.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836696 2026-02-20T17:06:32.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836696' 2026-02-20T17:06:32.804 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836696 2026-02-20T17:06:32.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:32.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836694 -lt 21474836696 2026-02-20T17:06:32.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:06:33.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:06:33.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:06:34.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836697 -lt 21474836696 2026-02-20T17:06:34.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:34.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673173 2026-02-20T17:06:34.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:34.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:06:34.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673173 2026-02-20T17:06:34.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:34.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673173 2026-02-20T17:06:34.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673173' 2026-02-20T17:06:34.127 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949673173 2026-02-20T17:06:34.127 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:06:34.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673174 -lt 42949673173 2026-02-20T17:06:34.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:06:34.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509651 2026-02-20T17:06:34.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:06:34.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:06:34.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509651 2026-02-20T17:06:34.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:06:34.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509651 2026-02-20T17:06:34.301 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509651 2026-02-20T17:06:34.301 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509651' 2026-02-20T17:06:34.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:06:34.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509652 -lt 64424509651 2026-02-20T17:06:34.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:06:34.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:06:34.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:06:34.606 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:06:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-20T17:06:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-20T17:06:34.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-20T17:06:39.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-20T17:06:39.626 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:04:36.717+0000 7f2a6f77d640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=16/17 n=1000 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [1,0,2] r=0 lpr=16 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep(periodic-regular)] scrubber: nodeep_scrub set, aborting 2026-02-20T17:06:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-20T17:06:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:39.627 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:39.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:39.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-20T17:06:39.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset nodeep-scrub 2026-02-20T17:06:40.143 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is unset 2026-02-20T17:06:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-20T17:06:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph osd unset noscrub 2026-02-20T17:06:40.344 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is unset 2026-02-20T17:06:40.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-20T17:06:40.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:40.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:40.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:40.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:41.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:41.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:41.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:41.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:41.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:41.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:41.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:41.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:41.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:42.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:42.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:42.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:42.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:42.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:42.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:42.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:42.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:42.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:43.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:43.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:43.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:43.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:43.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:43.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:43.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:43.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:43.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:44.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:44.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:44.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:44.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:44.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:44.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:44.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:45.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:45.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:46.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:46.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:46.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:46.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:46.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:46.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:46.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:46.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:46.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:47.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:47.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:47.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:47.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:47.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:47.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:47.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:47.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:47.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:48.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:48.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:48.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:48.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:48.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:48.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:48.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:48.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:49.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:49.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:49.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:49.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:49.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:49.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:49.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:49.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:49.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:50.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:50.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:50.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:50.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:50.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:50.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:50.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:50.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:50.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:51.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:51.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:51.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:51.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:51.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:51.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:51.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:52.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:52.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:53.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:53.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:53.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:53.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:53.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:53.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:53.137 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:53.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:53.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:54.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:54.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:54.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:54.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:54.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:54.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:54.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:54.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:54.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:55.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:55.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:55.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:55.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:55.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:55.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:55.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:55.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:55.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:56.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:56.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:56.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:56.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:56.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:56.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:56.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:56.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:56.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:57.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:57.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:57.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:57.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:57.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:57.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:57.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:57.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:57.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:06:58.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:06:58.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:06:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:06:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:06:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:06:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:06:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:06:59.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:06:59.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:00.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:00.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:00.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:00.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:00.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:00.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:00.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:00.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:00.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:01.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:01.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:01.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:01.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:01.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:01.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:01.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:01.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:01.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:02.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:02.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:02.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:02.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:02.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:02.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:02.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:02.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:02.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:03.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:03.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:03.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:03.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:03.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:03.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:03.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:03.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:03.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:04.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:04.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:04.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:04.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:04.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:04.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:04.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:04.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:04.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:05.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:05.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:05.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:05.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:05.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:05.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:05.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:06.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:06.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:07.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:07.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:07.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:07.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:07.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:07.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:07.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:07.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:07.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:08.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:08.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:08.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:08.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:08.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:08.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:08.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:08.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:08.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:09.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:09.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:09.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:09.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:09.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:09.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:09.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:09.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:09.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:10.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:10.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:10.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:10.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:10.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:10.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:10.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:10.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:10.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:11.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:11.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:11.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:11.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:11.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:11.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:11.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:11.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:11.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:12.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:12.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:12.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:12.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:12.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:12.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:12.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:12.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:12.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:13.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:13.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:13.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:13.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:13.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:13.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:13.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:14.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:14.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:15.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:15.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:15.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:15.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:15.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:16.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:16.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:16.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:16.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:16.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:17.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:17.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:17.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:17.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:17.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:17.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:17.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:17.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:17.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:18.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:18.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:18.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:18.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:18.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:18.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:18.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:18.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:18.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:19.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:19.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:19.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:19.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:19.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:19.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:19.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:19.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:20.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:20.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:20.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:20.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:20.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:20.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:20.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:21.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:21.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:22.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:22.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:22.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:22.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:22.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:23.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:23.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:23.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:23.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:23.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:23.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:23.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:23.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:23.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:24.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:24.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:24.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:24.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:24.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:24.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:24.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:24.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:24.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:25.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:25.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:25.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:25.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:25.590 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:25.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:25.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:25.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:25.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:26.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:26.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:26.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:26.749 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:26.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:26.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:26.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:26.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:27.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:27.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:27.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:27.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:27.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:27.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:27.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:28.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:28.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:29.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:29.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:29.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:29.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:29.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:29.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:29.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:29.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:29.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:30.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:30.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:30.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:30.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:30.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:30.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:30.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:30.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:30.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:31.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:31.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:31.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:31.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:31.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:31.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:31.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:31.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:31.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:32.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:32.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:32.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:32.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:32.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:32.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:32.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:32.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:32.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:33.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:33.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:33.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:33.719 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:33.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:33.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:33.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:33.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:33.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:34.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:34.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:34.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:34.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:35.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:35.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:36.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:36.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:36.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:36.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:36.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:36.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:36.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:36.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:36.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:37.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:37.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:37.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:37.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:37.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:37.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:38.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:38.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:38.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:38.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:38.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:38.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:38.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:38.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:38.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:39.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:39.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:39.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:39.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:39.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:39.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:39.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:39.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:39.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:40.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:40.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:40.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:40.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:40.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:40.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:40.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:40.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:40.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:41.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:41.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:41.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:41.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:41.840 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:41.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:41.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:41.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:41.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:42.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:42.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:42.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:42.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:43.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:43.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:43.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:43.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:43.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:44.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:44.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:44.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:44.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:44.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:44.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:44.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:44.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:45.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:45.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:45.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:45.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:45.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:45.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:45.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:45.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:45.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:46.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:46.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:46.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:46.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:46.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:46.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:46.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:46.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:46.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:47.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:47.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:47.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:47.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:47.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:47.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:47.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:47.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:47.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:48.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:48.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:48.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:48.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:48.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:48.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:48.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:48.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:49.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:49.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:49.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:49.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:49.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:49.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:49.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:50.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:50.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:51.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:51.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:51.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:51.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:51.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:51.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:51.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:51.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:51.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:52.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:52.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:52.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:52.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:52.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:52.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:52.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:52.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:52.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:53.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:53.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:53.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:53.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:53.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:53.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:53.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:53.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:53.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:54.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:54.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:54.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:54.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:54.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:54.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:54.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:55.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:55.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:55.738 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:55.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:55.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:55.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:55.739 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:55.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:55.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:56.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:56.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:56.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:56.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:56.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:56.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:56.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:57.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:57.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:58.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:58.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:58.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:58.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:58.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:58.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:58.052 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:58.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:58.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:07:59.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:07:59.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:07:59.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:07:59.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:07:59.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:07:59.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:07:59.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:07:59.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:07:59.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:00.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:00.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:00.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:00.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:00.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:00.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:00.360 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:00.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:00.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:01.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:01.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:01.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:01.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:01.513 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:01.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:01.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:01.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:01.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:02.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:02.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:02.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:02.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:02.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:02.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:02.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:02.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:02.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:03.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:03.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:03.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:03.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:03.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:03.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:03.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:03.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:03.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:04.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:04.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:04.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:04.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:04.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:04.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:04.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:05.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:05.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:06.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:06.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:06.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:06.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:06.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:06.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:06.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:06.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:06.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:07.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:07.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:07.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:07.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:07.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:07.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:08.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:08.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:08.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:08.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:08.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:09.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:09.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:09.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:09.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:09.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:09.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:09.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:09.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:09.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:10.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:10.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:10.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:10.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:10.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:10.791 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:10.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:10.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:11.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:11.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:11.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:11.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:11.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:11.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:11.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:12.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:12.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:13.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:13.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:13.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:13.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:13.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:13.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:13.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:13.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:13.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:14.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:14.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:14.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:14.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:14.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:14.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:14.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:14.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:14.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:15.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:15.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:15.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:15.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:15.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:16.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:16.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:16.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:16.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:16.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:16.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:16.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:16.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:16.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:17.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:17.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:17.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:17.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:17.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:17.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:17.748 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:17.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:17.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:18.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:18.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:18.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:18.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:18.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:18.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:18.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:19.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:19.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:20.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:20.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:20.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:20.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:20.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:20.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:20.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:20.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:20.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:21.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:21.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:21.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:21.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:21.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:21.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:21.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:21.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:21.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:22.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:22.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:22.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:22.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:22.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:22.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:22.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:22.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:22.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:23.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:23.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:23.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:23.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:23.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:23.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:23.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:23.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:23.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:24.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:24.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:24.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:24.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:25.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:25.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:25.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:25.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:25.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:25.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:25.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:26.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:26.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:27.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:27.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:27.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:27.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:27.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:27.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:27.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:27.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:27.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:28.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:28.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:28.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:28.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:28.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:28.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:28.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:28.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:28.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:29.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:29.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:29.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:29.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:29.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:29.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:29.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:29.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:29.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:30.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:30.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:30.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:30.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:30.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:30.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:30.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:30.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:30.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:31.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:31.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:31.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:31.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:31.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:31.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:31.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:31.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:31.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:32.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:32.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:32.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:32.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:32.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:33.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:33.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:33.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:33.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:33.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:33.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:33.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:34.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:34.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:35.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:35.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:35.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:35.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:35.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:35.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:35.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:35.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:35.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:36.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:36.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:36.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:36.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:36.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:36.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:36.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:37.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:37.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:37.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:37.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:37.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:37.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:37.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:37.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:37.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:38.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:38.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:38.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:38.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:38.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:38.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:38.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:38.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:39.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:39.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:39.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:39.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:39.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:39.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:39.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:39.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:39.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:40.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:40.913 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:40.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:40.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:40.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:40.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:40.914 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:41.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:41.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:42.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:42.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:42.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:42.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:42.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:42.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:42.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:42.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:42.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:43.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:43.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:43.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:43.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:43.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:43.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:44.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:44.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:44.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:44.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:44.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:44.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:44.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:44.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:44.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:45.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:45.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:45.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:45.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:45.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:45.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:45.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:45.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:45.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:46.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:46.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:46.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:46.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:46.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:46.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:46.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:46.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:46.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:47.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:47.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:47.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:47.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:47.843 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:47.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:47.844 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:48.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:48.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:49.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:49.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:49.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:49.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:49.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:49.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:49.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:49.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:49.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:50.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:50.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:50.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:50.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:50.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:50.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:50.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:50.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:50.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:51.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:51.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:51.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:51.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:51.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:51.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:51.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:51.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:51.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:52.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:52.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:52.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:52.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:52.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:52.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:52.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:52.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:52.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:53.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:53.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:53.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:53.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:54.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:54.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:54.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:54.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:54.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:54.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:54.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:54.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:55.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:55.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:55.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:55.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:55.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:55.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:55.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:56.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:56.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:57.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:57.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:57.112 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:57.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:57.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:57.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:57.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:57.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:57.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:58.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:58.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:58.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:58.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:58.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:58.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:58.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:58.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:58.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:08:59.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:08:59.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:08:59.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:08:59.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:08:59.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:08:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:08:59.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:08:59.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:08:59.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:00.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:00.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:00.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:00.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:00.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:00.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:00.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:00.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:00.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:01.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:01.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:01.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:01.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:01.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:01.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:01.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:01.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:01.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:02.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:02.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:02.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:02.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:02.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:02.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:02.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:03.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:03.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:04.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:04.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:04.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:04.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:04.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:04.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:04.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:04.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:04.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:05.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:05.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:05.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:06.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:06.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:06.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:06.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:06.373 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:06.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:06.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:06.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:06.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:07.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:07.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:07.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:07.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:07.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:07.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:07.529 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:07.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:07.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:08.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:08.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:08.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:08.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:08.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:08.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:08.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:08.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:08.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:09.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:09.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:09.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:09.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:09.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:09.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:09.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:10.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:10.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:11.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:11.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:11.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:11.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:11.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:11.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:11.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:11.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:11.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:12.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:12.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:12.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:12.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:12.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:12.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:12.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:12.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:12.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:13.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:13.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:13.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:13.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:13.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:13.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:13.319 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:13.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:13.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:14.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:14.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:14.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:14.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:14.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:15.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:15.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:15.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:15.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:16.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:16.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:16.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:16.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:16.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:16.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:16.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:16.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:16.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:17.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:17.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:17.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:17.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:18.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:18.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:19.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:19.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:19.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:19.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:19.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:19.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:19.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:19.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:19.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:20.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:20.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:20.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:20.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:20.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:20.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:20.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:20.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:20.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:21.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:21.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:21.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:21.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:21.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:21.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:21.428 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:21.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:21.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:22.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:22.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:22.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:22.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:22.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:22.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:22.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:22.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:22.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:23.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:23.746 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:23.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:23.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:24.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:24.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:24.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:24.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:24.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:24.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:24.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:25.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:25.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:26.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:26.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:26.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:26.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:26.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:26.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:26.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:26.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:26.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:27.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:27.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:27.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:27.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:27.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:27.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:27.226 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:27.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:27.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:28.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:28.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:28.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:28.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:28.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:28.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:28.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:28.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:28.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:29.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:29.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:29.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:29.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:29.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:29.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:29.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:29.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:29.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:30.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:30.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:30.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:30.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:30.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:30.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:30.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:30.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:30.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:31.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:31.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:31.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:31.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:31.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:31.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:31.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:32.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:32.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:33.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:33.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:33.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:33.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:33.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:33.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:33.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:33.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:34.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:34.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:34.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:34.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:34.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:34.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:34.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:34.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:34.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:35.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:35.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:35.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:35.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:35.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:35.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:35.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:35.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:36.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:36.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:36.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:36.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:36.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:36.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:36.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:36.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:36.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:37.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:37.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:37.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:37.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:37.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:37.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:37.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:37.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:37.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:38.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:38.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:38.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:38.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:38.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:38.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:38.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:38.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:38.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:39.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:39.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:39.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:39.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:39.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:39.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:39.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:40.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:40.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:41.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:41.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:41.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:41.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:41.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:41.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:41.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:41.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:41.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:42.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:42.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:42.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:42.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:42.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:42.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:42.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:42.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:42.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:43.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:43.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:43.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:43.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:43.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:43.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:44.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:44.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:44.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:44.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:44.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:44.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:44.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:44.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:45.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:45.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:45.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:45.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:45.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:45.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:45.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:45.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:45.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:46.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:46.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:46.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:46.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:46.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:46.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:46.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:47.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:47.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:48.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:48.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:48.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:48.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:48.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:48.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:48.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:48.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:48.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:49.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:49.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:49.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:49.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:49.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:49.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:49.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:49.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:49.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:50.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:50.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:50.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:50.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:50.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:50.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:50.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:50.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:50.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:51.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:51.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:51.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:51.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:51.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:51.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:51.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:51.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:51.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:52.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:52.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:52.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:52.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:52.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:52.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:52.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:52.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:53.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:53.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:53.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:53.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:53.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:53.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:53.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:54.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:54.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:55.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:55.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:55.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:55.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:55.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:55.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:55.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:55.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:55.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:56.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:56.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:56.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:56.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:56.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:56.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:56.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:56.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:56.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:57.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:57.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:57.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:57.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:57.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:57.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:57.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:57.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:57.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:58.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:58.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:58.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:58.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:58.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:58.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:58.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:58.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:58.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:09:59.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:09:59.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:09:59.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:09:59.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:09:59.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:09:59.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:09:59.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:09:59.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:09:59.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:00.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:00.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:00.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:00.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:00.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:00.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:00.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:00.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:00.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:01.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:01.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:01.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:01.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:01.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:01.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:01.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:02.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:02.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:03.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:03.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:03.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:03.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:03.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:03.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:03.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:03.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:04.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:04.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:04.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:04.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:04.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:04.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:04.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:04.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:04.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:05.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:05.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:05.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:05.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:05.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:05.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:05.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:05.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:05.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:06.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:06.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:06.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:06.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:06.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:06.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:06.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:06.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:06.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:07.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:07.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:07.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:07.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:07.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:07.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:07.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:07.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:08.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:08.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:08.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:08.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:08.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:08.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:08.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:09.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:09.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:10.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:10.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:10.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:10.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:10.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:10.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:10.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:10.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:11.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:11.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:11.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:11.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:11.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:11.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:11.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:11.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:11.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:12.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:12.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:12.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:12.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:12.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:12.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:12.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:12.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:12.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:13.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:13.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:13.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:13.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:13.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:13.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:13.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:13.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:13.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:14.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:14.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:14.709 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:14.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:14.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:14.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:14.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:15.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:15.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:15.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:15.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:15.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:15.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:15.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:16.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:16.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:17.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:17.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:17.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:17.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:17.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:17.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:17.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:17.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:17.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:18.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:18.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:18.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:18.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:18.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:18.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:18.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:18.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:18.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:19.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:19.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:19.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:19.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:19.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:19.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:19.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:19.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:19.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:20.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:20.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:20.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:20.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:20.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:20.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:21.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:21.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:21.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:21.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:21.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:21.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:21.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:21.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:21.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:22.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:22.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:22.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:22.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:22.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:22.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:22.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:22.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:22.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:23.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:23.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:23.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:23.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:23.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:23.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:23.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:24.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:24.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:25.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:25.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:25.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:25.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:25.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:25.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:25.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:25.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:25.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:26.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:26.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:26.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:26.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:26.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:26.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:26.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:26.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:26.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:27.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:27.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:27.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:27.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:27.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:27.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:27.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:27.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:27.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:28.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:28.593 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:28.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:28.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:28.594 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:28.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:28.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:28.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:28.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:29.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:29.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:29.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:29.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:29.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:29.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:29.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:29.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:29.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:30.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:30.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:30.911 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:30.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:30.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:30.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:30.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:31.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:31.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:32.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:32.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:32.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:32.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:32.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:32.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:32.073 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:32.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:32.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:33.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:33.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:33.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:33.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:33.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:33.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:33.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:33.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:33.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:34.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:34.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:34.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:34.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:34.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:34.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:34.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:34.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:34.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:35.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:35.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:35.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:35.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:35.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:35.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:35.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:35.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:35.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:36.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:36.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:36.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:36.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:36.710 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:36.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:36.711 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:36.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:36.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:37.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:37.869 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:37.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:37.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:37.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:37.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:37.870 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:38.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:38.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:39.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:39.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:39.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:39.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:39.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:39.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:39.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:39.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:39.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:40.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:40.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:40.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:40.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:40.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:40.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:40.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:40.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:40.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:41.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:41.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:41.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:41.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:41.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:41.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:41.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:41.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:41.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:42.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:42.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:42.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:42.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:43.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:43.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:43.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:43.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:43.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:44.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:44.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:44.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:44.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:44.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:44.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:44.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:44.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:44.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:45.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:45.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:45.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:45.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:45.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:45.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:45.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:46.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:46.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:47.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:47.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:47.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:47.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:47.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:47.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:47.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:47.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:47.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:48.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:48.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:48.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:48.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:48.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:48.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:48.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:48.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:48.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:49.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:49.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:49.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:49.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:49.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:49.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:49.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:49.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:49.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:50.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:50.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:50.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:50.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:50.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:50.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:50.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:50.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:50.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:51.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:51.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:51.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:51.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:51.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:51.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:51.765 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:51.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:51.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:52.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:52.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:52.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:52.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:52.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:52.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:52.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:53.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:53.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:54.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:54.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:54.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:54.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:54.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:55.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:55.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:55.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:55.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:55.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:55.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:55.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:55.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:55.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:56.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:56.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:56.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:56.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:56.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:56.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:56.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:56.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:56.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:57.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:57.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:57.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:57.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:57.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:57.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:57.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:57.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:57.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:58.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:58.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:58.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:58.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:58.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:58.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:58.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:10:58.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:10:58.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:10:59.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:10:59.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:10:59.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:10:59.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:10:59.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:10:59.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:10:59.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:00.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:01.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:01.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:01.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:01.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:01.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:01.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:01.020 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:01.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:01.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:02.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:02.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:02.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:02.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:02.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:02.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:02.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:02.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:02.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:03.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:03.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:03.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:03.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:03.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:03.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:03.338 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:03.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:03.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:04.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:04.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:04.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:04.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:04.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:04.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:04.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:04.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:04.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:05.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:05.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:05.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:05.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:05.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:05.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:05.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:05.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:05.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:06.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:06.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:06.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:06.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:06.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:06.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:06.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:06.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:06.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:07.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:07.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:07.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:07.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:07.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:07.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:07.971 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:08.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:08.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:09.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:09.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:09.131 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:09.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:09.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:09.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:10.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:10.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:10.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:10.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:10.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:10.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:10.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:10.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:10.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:11.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:11.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:11.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:11.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:11.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:11.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:12.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:12.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:12.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:12.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:12.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:12.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:12.615 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:12.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:12.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:13.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:13.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:13.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:13.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:13.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:13.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:13.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:13.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:13.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:14.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:14.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:15.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:15.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:16.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:16.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:16.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:16.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:16.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:17.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:17.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:17.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:17.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:17.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:17.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:17.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:17.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:17.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:18.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:18.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:18.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:18.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:18.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:18.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:18.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:18.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:19.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:19.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:19.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:19.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:19.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:19.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:19.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:19.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:19.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:20.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:20.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:20.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:20.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:20.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:20.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:20.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:20.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:20.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:21.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:21.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:21.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:21.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:21.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:21.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:21.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:22.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:22.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:23.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:23.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:23.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:23.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:23.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:23.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:23.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:23.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:23.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:24.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:24.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:24.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:24.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:24.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:24.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:24.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:25.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:25.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:25.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:25.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:25.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:25.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:25.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:25.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:25.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:26.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:26.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:27.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:27.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:27.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:27.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:27.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:27.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:27.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:27.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:27.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:28.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:28.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:28.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:28.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:28.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:28.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:28.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:28.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:28.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:29.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:29.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:29.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:29.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:29.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:29.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:29.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:30.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:30.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:31.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:31.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:31.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:31.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:31.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:31.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:31.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:31.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:31.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:32.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:32.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:32.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:32.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:32.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:32.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:32.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:32.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:32.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:33.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:33.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:33.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:33.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:33.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:33.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:33.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:33.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:33.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:34.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:34.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:34.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:34.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:34.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:35.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:35.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:35.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:35.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:35.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:35.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:35.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:35.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:35.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:36.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:36.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:36.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:36.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:36.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:36.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:36.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:37.079 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:37.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:38.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:38.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:38.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:38.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:38.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:38.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:38.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:38.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:38.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:39.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:39.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:39.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:39.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:39.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:39.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:39.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:39.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:39.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:40.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:40.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:40.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:40.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:40.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:40.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:40.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:40.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:40.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:41.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:41.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:41.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:41.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:41.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:42.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:42.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:42.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:42.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:42.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:42.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:42.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:42.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:42.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:43.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:43.879 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:43.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:43.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:43.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:43.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:43.880 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:44.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:44.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:45.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:45.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:45.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:45.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:45.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:45.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:45.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:45.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:45.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:46.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:46.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:46.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:46.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:46.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:46.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:46.199 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:46.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:46.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:47.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:47.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:47.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:47.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:47.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:47.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:47.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:47.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:47.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:48.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:48.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:48.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:48.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:48.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:48.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:48.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:48.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:48.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:49.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:49.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:49.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:49.676 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:49.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:49.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:49.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:49.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:50.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:50.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:50.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:50.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:50.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:50.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:50.837 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:50.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:50.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:51.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:51.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:51.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:51.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:51.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:51.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:51.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:52.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:52.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:53.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:53.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:53.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:53.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:53.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:53.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:53.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:53.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:53.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:54.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:54.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:54.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:54.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:54.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:54.312 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:54.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:54.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:54.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:55.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:55.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:55.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:55.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:55.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:55.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:55.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:55.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:55.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:56.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:56.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:56.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:56.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:56.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:57.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:57.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:57.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:57.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:57.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:57.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:57.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:57.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:11:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:11:58.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:11:58.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:11:58.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:11:58.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:11:58.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:11:58.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:11:59.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:11:59.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:00.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:00.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:00.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:00.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:00.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:00.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:00.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:00.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:00.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:01.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:01.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:01.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:01.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:02.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:02.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:02.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:02.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:02.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:02.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:02.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:02.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:02.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:03.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:03.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:03.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:03.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:03.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:03.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:03.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:03.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:03.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:04.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:04.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:04.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:04.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:05.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:05.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:05.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:05.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:05.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:05.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:06.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:06.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:07.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:07.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:07.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:07.057 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:07.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:07.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:07.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:08.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:08.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:08.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:08.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:08.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:08.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:08.212 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:08.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:08.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:09.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:09.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:09.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:09.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:09.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:09.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:09.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:09.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:09.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:10.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:10.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:10.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:10.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:10.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:10.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:10.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:10.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:10.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:11.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:11.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:11.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:11.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:11.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:11.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:11.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:11.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:11.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:12.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:12.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:12.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:12.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:12.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:12.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:12.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:13.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:13.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:13.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:13.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:13.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:13.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:13.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:14.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:14.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:15.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:15.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:15.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:15.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:15.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:15.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:15.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:15.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:15.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:16.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:16.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:16.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:16.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:16.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:16.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:16.316 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:16.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:17.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:17.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:17.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:17.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:17.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:17.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:17.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:17.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:17.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:18.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:18.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:18.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:18.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:18.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:18.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:18.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:18.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:18.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:19.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:19.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:19.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:19.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:19.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:19.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:19.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:19.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:19.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:20.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:20.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:20.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:20.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:20.952 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:20.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:20.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:21.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:21.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:22.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:22.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:22.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:22.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:22.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:23.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:23.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:23.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:23.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:23.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:23.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:23.266 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:23.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:23.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:24.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:24.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:24.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:24.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:24.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:24.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:24.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:24.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:24.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:25.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:25.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:25.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:25.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:25.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:25.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:25.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:25.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:25.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:26.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:26.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:26.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:26.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:26.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:26.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:26.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:26.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:26.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:27.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:27.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:27.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:27.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:27.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:27.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:27.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:28.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:28.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:29.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:29.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:29.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:29.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:29.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:29.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:29.039 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:29.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:29.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:30.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:30.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:30.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:30.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:30.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:30.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:30.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:30.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:30.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:31.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:31.347 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:31.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:31.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:31.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:31.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:31.348 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:31.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:31.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:32.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:32.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:32.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:32.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:32.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:32.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:32.502 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:32.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:32.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:33.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:33.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:33.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:33.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:33.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:33.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:33.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:33.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:33.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:34.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:34.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:34.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:34.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:34.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:34.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:34.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:34.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:35.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:35.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:35.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:35.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:35.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:35.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:36.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:36.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:37.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:37.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:37.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:37.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:37.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:37.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:37.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:37.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:37.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:38.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:38.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:38.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:38.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:38.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:38.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:38.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:38.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:38.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:39.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:39.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:39.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:39.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:39.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:39.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:39.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:39.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:39.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:40.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:40.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:40.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:40.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:40.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:40.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:40.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:40.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:40.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:41.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:41.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:41.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:41.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:41.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:41.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:41.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:41.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:41.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:42.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:42.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:42.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:42.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:42.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:42.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:42.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:43.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:43.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:44.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:44.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:44.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:44.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:44.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:44.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:44.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:44.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:44.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:45.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:45.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:45.249 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:45.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:45.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:46.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:46.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:46.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:46.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:46.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:46.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:46.408 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:46.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:46.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:47.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:47.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:47.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:47.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:47.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:47.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:47.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:47.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:47.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:48.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:48.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:48.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:48.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:48.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:48.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:48.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:48.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:49.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:49.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:49.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:49.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:49.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:49.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:49.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:50.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:50.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:51.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:51.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:51.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:51.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:51.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:51.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:51.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:51.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:04:01.897604+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:51.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:12:52.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:12:52.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:12:52.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:12:52.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:12:52.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:12:52.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:12:52.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:12:52.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:12:49.860638+0000 '>' 2026-02-20T17:04:01.897604+0000 2026-02-20T17:12:52.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:12:52.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-20T17:12:52.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:12:52.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:12:52.362 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:12:52.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:12:52.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:52.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:12:52.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.438 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.439 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.440 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.441 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.442 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.443 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.443 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.443 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:12:52.443 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:12:52.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:52.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:12:52.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:12:52.507 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:12:52.507 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:12:52.507 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.507 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.508 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.509 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 67, 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.510 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.511 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.512 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.513 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.513 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:12:52.513 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:12:52.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:52.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:12:52.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.578 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.579 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.580 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:12:52.581 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:12:52.582 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:12:52.583 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:12:52.583 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2026-02-20T17:12:52.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2026-02-20T17:12:52.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:12:52.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:12:52.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:12:52.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:12:52.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:12:52.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:12:52.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:12:52.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:12:52.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:12:52.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:12:52.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:12:52.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:12:52.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:12:52.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:12:52.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:12:52.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:12:52.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:12:52.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:12:52.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:52.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:52.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:12:52.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:12:52.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:12:52.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2026-02-20T17:12:52.961 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2026-02-20T17:12:52.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:12:52.962 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2026-02-20T17:12:52.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:12:52.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:12:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:12:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:12:52.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:12:52.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:12:52.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:12:52.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:12:52.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:12:52.967 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:12:52.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:12:52.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:12:52.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:12:52.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:12:52.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:12:52.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:12:52.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:12:52.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:12:52.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:52.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:52.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:12:52.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:12:52.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:12:52.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:12:52.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:12:52.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:52.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:52.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:12:52.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:12:52.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:12:52.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:12:52.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:12:52.978 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local poolname=test 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: local OSDS=3 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: local objects=90 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: TESTDATA=testdata.515468 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:12:52.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:12:52.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:12:52.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:12:52.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:12:52.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:12:52.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:53.000 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:53.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:12:53.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:12:53.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:12:53.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:12:53.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:12:53.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:12:53.029 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:12:53.030 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:12:53.030 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:12:53.030 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:12:53.030 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:12:53.031 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:53.031 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:53.031 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:12:53.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:12:53.031 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:12:53.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:12:53.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:12:53.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:12:53.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:12:53.085 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:12:53.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:12:53.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:12:53.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:12:53.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:538: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:12:53.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:53.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:53.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:12:53.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:12:53.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:552: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-20T17:12:53.249 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: expr 3 - 1 2026-02-20T17:12:53.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: seq 0 2 2026-02-20T17:12:53.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:53.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:12:53.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:12:53.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:12:53.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:12:53.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-20T17:12:53.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:12:53.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:12:53.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2516dd15-f875-4424-bb6c-44ba076e25a3 2026-02-20T17:12:53.256 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 2516dd15-f875-4424-bb6c-44ba076e25a3 2026-02-20T17:12:53.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2516dd15-f875-4424-bb6c-44ba076e25a3' 2026-02-20T17:12:53.257 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:12:53.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVlphpw5jNDxAADv6PkcTfoAeeywQnnRY64w== 2026-02-20T17:12:53.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVlphpw5jNDxAADv6PkcTfoAeeywQnnRY64w=="}' 2026-02-20T17:12:53.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2516dd15-f875-4424-bb6c-44ba076e25a3 -i td/osd-scrub-test/0/new.json 2026-02-20T17:12:53.358 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:12:53.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:12:53.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQAVlphpw5jNDxAADv6PkcTfoAeeywQnnRY64w== --osd-uuid 2516dd15-f875-4424-bb6c-44ba076e25a3 2026-02-20T17:12:53.385 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:53.382+0000 7f607d617a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:53.386 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:53.382+0000 7f607d617a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:53.387 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:53.382+0000 7f607d617a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:53.388 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:53.382+0000 7f607d617a40 -1 bdev(0x559fdf760c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:12:53.388 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:53.382+0000 7f607d617a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:12:54.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:12:54.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:12:54.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:12:54.185 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:12:54.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:12:54.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:12:54.286 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:12:54.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:12:54.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:12:54.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:12:54.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:12:54.314 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:54.310+0000 7f1e16acda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:54.316 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:54.314+0000 7f1e16acda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:54.317 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:54.314+0000 7f1e16acda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:12:54.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:12:54.378 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:12:54.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:12:54.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:12:54.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:12:54.830 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:54.826+0000 7f1e16acda40 -1 Falling back to public interface 2026-02-20T17:12:55.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:12:55.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:12:55.473 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:12:55.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:12:55.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:12:55.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:12:55.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:12:55.758 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:55.754+0000 7f1e16acda40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:12:56.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:12:56.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:12:56.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:12:56.564 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:12:56.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:12:56.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:12:56.717 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4104988941,v1:127.0.0.1:6801/4104988941] [v2:127.0.0.1:6802/4104988941,v1:127.0.0.1:6803/4104988941] exists,up 2516dd15-f875-4424-bb6c-44ba076e25a3 2026-02-20T17:12:56.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:12:56.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:12:56.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:56.722 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:12:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-20T17:12:56.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:12:56.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:12:56.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d78ce7d-c523-4933-8fe9-a52acbf7a39e 2026-02-20T17:12:56.724 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 8d78ce7d-c523-4933-8fe9-a52acbf7a39e 2026-02-20T17:12:56.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8d78ce7d-c523-4933-8fe9-a52acbf7a39e' 2026-02-20T17:12:56.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:12:56.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAYlphpJBGoKxAA5xU/sYwvj51piqPqFpsYZg== 2026-02-20T17:12:56.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAYlphpJBGoKxAA5xU/sYwvj51piqPqFpsYZg=="}' 2026-02-20T17:12:56.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d78ce7d-c523-4933-8fe9-a52acbf7a39e -i td/osd-scrub-test/1/new.json 2026-02-20T17:12:56.894 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:12:56.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:12:56.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQAYlphpJBGoKxAA5xU/sYwvj51piqPqFpsYZg== --osd-uuid 8d78ce7d-c523-4933-8fe9-a52acbf7a39e 2026-02-20T17:12:56.931 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:56.926+0000 7f64f3547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:56.932 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:56.930+0000 7f64f3547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:56.933 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:56.930+0000 7f64f3547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:56.934 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:56.930+0000 7f64f3547a40 -1 bdev(0x55a8f4758c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:12:56.934 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:56.930+0000 7f64f3547a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:12:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:12:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:12:57.557 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:12:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:12:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:12:57.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:12:57.749 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:12:57.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:12:57.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:12:57.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:12:57.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:12:57.776 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:57.770+0000 7f42b0781a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:57.777 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:57.774+0000 7f42b0781a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:57.779 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:57.774+0000 7f42b0781a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:12:57.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:12:57.911 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:12:57.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:12:57.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:12:58.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:12:58.162 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:58.158+0000 7f42b0781a40 -1 Falling back to public interface 2026-02-20T17:12:58.818 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:58.814+0000 7f42b0781a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:12:59.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:12:59.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:12:59.073 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:12:59.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:12:59.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:12:59.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:12:59.227 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/950521967,v1:127.0.0.1:6811/950521967] [v2:127.0.0.1:6812/950521967,v1:127.0.0.1:6813/950521967] exists,up 8d78ce7d-c523-4933-8fe9-a52acbf7a39e 2026-02-20T17:12:59.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:12:59.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:12:59.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:12:59.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:12:59.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:12:59.233 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:12:59.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:12:59.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e8ad7368-59a8-4573-a326-90e9d37a03ae 2026-02-20T17:12:59.235 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 e8ad7368-59a8-4573-a326-90e9d37a03ae 2026-02-20T17:12:59.236 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e8ad7368-59a8-4573-a326-90e9d37a03ae' 2026-02-20T17:12:59.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:12:59.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAblphpJsaGDhAAnI3jOk2Z+t8peirKXFPLLg== 2026-02-20T17:12:59.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAblphpJsaGDhAAnI3jOk2Z+t8peirKXFPLLg=="}' 2026-02-20T17:12:59.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e8ad7368-59a8-4573-a326-90e9d37a03ae -i td/osd-scrub-test/2/new.json 2026-02-20T17:12:59.397 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:12:59.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:12:59.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQAblphpJsaGDhAAnI3jOk2Z+t8peirKXFPLLg== --osd-uuid e8ad7368-59a8-4573-a326-90e9d37a03ae 2026-02-20T17:12:59.436 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:59.430+0000 7fdd9e733a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:59.437 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:59.434+0000 7fdd9e733a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:59.438 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:59.434+0000 7fdd9e733a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:12:59.439 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:59.434+0000 7fdd9e733a40 -1 bdev(0x55f9bc84ac00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:12:59.439 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:12:59.434+0000 7fdd9e733a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:13:00.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:13:00.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:13:00.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:13:00.153 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:13:00.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:13:00.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:13:00.359 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:13:00.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-20T17:13:00.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:13:00.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:13:00.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:13:00.376 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:00.374+0000 7f7faf35fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:00.378 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:00.374+0000 7f7faf35fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:00.379 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:00.374+0000 7f7faf35fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:00.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:13:00.520 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:13:00.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:00.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:13:00.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:13:00.842 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:00.838+0000 7f7faf35fa40 -1 Falling back to public interface 2026-02-20T17:13:01.455 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:01.450+0000 7f7faf35fa40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:01.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1478628824,v1:127.0.0.1:6819/1478628824] [v2:127.0.0.1:6820/1478628824,v1:127.0.0.1:6821/1478628824] exists,up e8ad7368-59a8-4573-a326-90e9d37a03ae 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: create_pool test 1 1 2026-02-20T17:13:01.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:13:02.016 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:13:02.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:13:03.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:561: TEST_dump_scrub_schedule: wait_for_clean 2026-02-20T17:13:03.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:13:03.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:13:03.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:13:03.030 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:13:03.031 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:13:03.031 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:13:03.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:13:03.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:13:03.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:13:03.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:13:03.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:13:03.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:13:03.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:13:03.126 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:13:03.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:13:03.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:13:03.282 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:13:03.282 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:13:03.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:13:03.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:13:03.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:13:03.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T17:13:03.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T17:13:03.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T17:13:03.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:13:03.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:13:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-20T17:13:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-20T17:13:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2026-02-20T17:13:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:13:03.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:13:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-20T17:13:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-20T17:13:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-60129542146' 2026-02-20T17:13:03.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:13:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T17:13:03.504 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:13:03.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:13:03.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T17:13:03.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:13:03.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T17:13:03.508 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T17:13:03.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T17:13:03.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:13:03.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T17:13:03.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:13:03.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-20T17:13:03.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:13:03.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:13:03.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-20T17:13:03.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:13:03.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-20T17:13:03.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-20T17:13:03.666 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672965 2026-02-20T17:13:03.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:13:03.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-20T17:13:03.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:13:03.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-20T17:13:03.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:13:03.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:13:03.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-20T17:13:03.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:13:03.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-20T17:13:03.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-20T17:13:03.832 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542146 2026-02-20T17:13:03.833 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:13:03.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542145 -lt 60129542146 2026-02-20T17:13:03.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:13:04.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:13:04.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:13:05.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542145 -lt 60129542146 2026-02-20T17:13:05.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:13:06.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:13:06.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:13:06.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542146 2026-02-20T17:13:06.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:13:06.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:13:06.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:13:06.505 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:13:06.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:13:06.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:13:06.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:13:06.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:13:06.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:13:06.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:13:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:13:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:13:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:13:06.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: ceph osd dump 2026-02-20T17:13:06.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2026-02-20T17:13:06.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: awk '{ print $2 }' 2026-02-20T17:13:07.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: poolid=1 2026-02-20T17:13:07.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:13:07.009 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:13:07.009 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:13:07.009 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5786e-05 s, 28.8 MB/s 2026-02-20T17:13:07.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: seq 1 90 2026-02-20T17:13:07.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.515468 2026-02-20T17:13:07.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.515468 2026-02-20T17:13:07.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.515468 2026-02-20T17:13:07.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.515468 2026-02-20T17:13:07.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.515468 2026-02-20T17:13:07.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.515468 2026-02-20T17:13:07.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.515468 2026-02-20T17:13:07.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.515468 2026-02-20T17:13:07.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.515468 2026-02-20T17:13:07.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.515468 2026-02-20T17:13:07.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.515468 2026-02-20T17:13:07.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.515468 2026-02-20T17:13:07.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.515468 2026-02-20T17:13:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.515468 2026-02-20T17:13:07.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.515468 2026-02-20T17:13:07.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj16 testdata.515468 2026-02-20T17:13:07.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj17 testdata.515468 2026-02-20T17:13:07.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj18 testdata.515468 2026-02-20T17:13:07.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj19 testdata.515468 2026-02-20T17:13:07.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj20 testdata.515468 2026-02-20T17:13:07.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj21 testdata.515468 2026-02-20T17:13:07.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj22 testdata.515468 2026-02-20T17:13:07.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj23 testdata.515468 2026-02-20T17:13:07.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj24 testdata.515468 2026-02-20T17:13:07.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj25 testdata.515468 2026-02-20T17:13:07.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj26 testdata.515468 2026-02-20T17:13:07.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj27 testdata.515468 2026-02-20T17:13:07.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj28 testdata.515468 2026-02-20T17:13:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj29 testdata.515468 2026-02-20T17:13:07.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj30 testdata.515468 2026-02-20T17:13:07.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj31 testdata.515468 2026-02-20T17:13:07.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj32 testdata.515468 2026-02-20T17:13:07.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj33 testdata.515468 2026-02-20T17:13:07.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj34 testdata.515468 2026-02-20T17:13:07.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj35 testdata.515468 2026-02-20T17:13:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj36 testdata.515468 2026-02-20T17:13:07.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj37 testdata.515468 2026-02-20T17:13:07.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj38 testdata.515468 2026-02-20T17:13:07.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj39 testdata.515468 2026-02-20T17:13:07.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj40 testdata.515468 2026-02-20T17:13:07.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj41 testdata.515468 2026-02-20T17:13:07.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj42 testdata.515468 2026-02-20T17:13:07.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj43 testdata.515468 2026-02-20T17:13:07.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj44 testdata.515468 2026-02-20T17:13:07.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:07.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj45 testdata.515468 2026-02-20T17:13:08.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj46 testdata.515468 2026-02-20T17:13:08.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj47 testdata.515468 2026-02-20T17:13:08.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj48 testdata.515468 2026-02-20T17:13:08.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj49 testdata.515468 2026-02-20T17:13:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj50 testdata.515468 2026-02-20T17:13:08.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj51 testdata.515468 2026-02-20T17:13:08.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj52 testdata.515468 2026-02-20T17:13:08.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj53 testdata.515468 2026-02-20T17:13:08.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj54 testdata.515468 2026-02-20T17:13:08.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj55 testdata.515468 2026-02-20T17:13:08.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj56 testdata.515468 2026-02-20T17:13:08.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj57 testdata.515468 2026-02-20T17:13:08.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj58 testdata.515468 2026-02-20T17:13:08.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj59 testdata.515468 2026-02-20T17:13:08.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj60 testdata.515468 2026-02-20T17:13:08.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj61 testdata.515468 2026-02-20T17:13:08.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj62 testdata.515468 2026-02-20T17:13:08.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj63 testdata.515468 2026-02-20T17:13:08.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj64 testdata.515468 2026-02-20T17:13:08.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj65 testdata.515468 2026-02-20T17:13:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj66 testdata.515468 2026-02-20T17:13:08.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj67 testdata.515468 2026-02-20T17:13:08.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj68 testdata.515468 2026-02-20T17:13:08.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj69 testdata.515468 2026-02-20T17:13:08.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj70 testdata.515468 2026-02-20T17:13:08.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj71 testdata.515468 2026-02-20T17:13:08.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj72 testdata.515468 2026-02-20T17:13:08.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj73 testdata.515468 2026-02-20T17:13:08.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj74 testdata.515468 2026-02-20T17:13:08.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj75 testdata.515468 2026-02-20T17:13:08.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj76 testdata.515468 2026-02-20T17:13:08.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj77 testdata.515468 2026-02-20T17:13:08.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj78 testdata.515468 2026-02-20T17:13:08.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj79 testdata.515468 2026-02-20T17:13:08.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj80 testdata.515468 2026-02-20T17:13:08.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj81 testdata.515468 2026-02-20T17:13:08.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj82 testdata.515468 2026-02-20T17:13:08.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj83 testdata.515468 2026-02-20T17:13:08.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj84 testdata.515468 2026-02-20T17:13:08.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj85 testdata.515468 2026-02-20T17:13:08.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj86 testdata.515468 2026-02-20T17:13:08.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj87 testdata.515468 2026-02-20T17:13:08.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:08.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj88 testdata.515468 2026-02-20T17:13:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:09.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj89 testdata.515468 2026-02-20T17:13:09.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-20T17:13:09.038 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj90 testdata.515468 2026-02-20T17:13:09.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:569: TEST_dump_scrub_schedule: rm -f testdata.515468 2026-02-20T17:13:09.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:571: TEST_dump_scrub_schedule: local pgid=1.0 2026-02-20T17:13:09.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2026-02-20T17:13:09.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: local now_is=2026-02-20T17:13:09.066442088+00:00 2026-02-20T17:13:09.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2026-02-20T17:13:09.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: declare -A expct_starting 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:582: TEST_dump_scrub_schedule: declare -A sched_data 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:583: TEST_dump_scrub_schedule: extract_published_sch 1.0 2026-02-20T17:13:09.066442088+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-20T17:13:09.066442088+00:00 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-20T17:13:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-20T17:13:09.209 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 0, 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:13:02.013843+0000", 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:13:09.219 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:09.220 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 17, 2026-02-20T17:13:09.220 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 16 2026-02-20T17:13:09.220 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:09.220 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:13:02.013843+0000", 2026-02-20T17:13:09.292 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:13:02.013 (2026-02-21T17:13:02.013)" 2026-02-20T17:13:09.293 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:09.293 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:09.375 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:09.375 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=17 2026-02-20T17:13:09.375 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=108 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:13:02.013' 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:13:02.013' 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=0 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:13:02.013843+0000' 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='0x0' 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:13:09.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=0 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:13:02.013843+0000' 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=17 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=16 2026-02-20T17:13:09.377 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:09.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2026-02-20T17:13:09.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2026-02-20T17:13:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2026-02-20T17:13:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2026-02-20T17:13:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2026-02-20T17:13:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stdout:-- - comparing: 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stdout:key is query_is_future expected: true in actual: true 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stdout:key is query_active expected: false in actual: false 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2026-02-20T17:13:09.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: (( 0 == 0 )) 2026-02-20T17:13:09.389 INFO:tasks.workunit.client.0.trial013.stdout:last-scrub --- 0x0 2026-02-20T17:13:09.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2026-02-20T17:13:09.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-20T17:13:02.013843+0000 2026-02-20T17:13:09.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2026-02-20T17:13:09.448 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:13:09.448 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:09.448 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:09.455 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:13:09.455 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:09.455 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:09.462 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:13:09.462 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:09.463 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:09.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2026-02-20T17:13:09.543 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:09.543 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T17:13:09.544 INFO:tasks.workunit.client.0.trial013.stdout: "must": true, 2026-02-20T17:13:09.544 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "0.000000" 2026-02-20T17:13:09.544 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:09.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sleep 5 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sched_data=() 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:602: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-20T17:13:02.013843+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T17:13:02.013843+0000 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-20T17:13:14.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-20T17:13:14.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:13:14.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:13:14.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:13:14.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:13:14.554 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2026-02-20T17:13:02.013843+0000 (10 retries) 2026-02-20T17:13:15.207 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:13:09.905635+0000", 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:13:15.217 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:15.218 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 17, 2026-02-20T17:13:15.218 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 123 2026-02-20T17:13:15.218 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:15.218 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:15.287 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:13:09.905635+0000", 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:13:09.905 (2026-02-21T17:13:09.905)" 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:15.288 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:15.375 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:15.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=17 2026-02-20T17:13:15.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=125 2026-02-20T17:13:15.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:13:15.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:13:15.376 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:13:09.905' 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:13:09.905' 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:13:09.905635+0000' 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:13:15.377 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:13:15.378 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:15.378 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:15.378 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:15.378 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:15.378 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:13:09.905635+0000' 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=17 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=123 2026-02-20T17:13:15.379 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:15.388 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ false / 125 / 123 / true / 2026-02-20T17:13:09.905635+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-20T17:13:15.388 INFO:tasks.workunit.client.0.trial013.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-20T17:13:15.388 INFO:tasks.workunit.client.0.trial013.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-20T17:13:15.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:13:15.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:604: TEST_dump_scrub_schedule: sched_data=() 2026-02-20T17:13:15.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:606: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-20T17:13:02.013843+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T17:13:02.013843+0000 2026-02-20T17:13:15.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2026-02-20T17:13:15.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-20T17:13:15.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:13:15.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:13:15.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:13:15.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:13:15.394 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2026-02-20T17:13:02.013843+0000 (10 retries) 2026-02-20T17:13:16.040 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:16.050 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:16.050 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:16.050 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:16.050 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:13:09.905635+0000", 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 17, 2026-02-20T17:13:16.051 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 124 2026-02-20T17:13:16.052 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:16.052 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:13:09.905635+0000", 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:13:09.905 (2026-02-21T17:13:09.905)" 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:16.120 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=17 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=125 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:13:09.905' 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:13:09.905' 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:13:09.905635+0000' 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:13:16.207 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:13:09.905635+0000' 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=17 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=124 2026-02-20T17:13:16.208 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:16.217 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ false / 125 / 124 / true / 2026-02-20T17:13:09.905635+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2026-02-20T17:13:16.217 INFO:tasks.workunit.client.0.trial013.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-20T17:13:16.217 INFO:tasks.workunit.client.0.trial013.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-20T17:13:16.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:13:16.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph osd set noscrub 2026-02-20T17:13:16.404 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:13:16.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 2 2026-02-20T17:13:18.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-20T17:13:18.483 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:13:18.483 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:13:18.483 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.490 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:13:18.490 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:13:18.491 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.497 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:13:18.498 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-20T17:13:18.498 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2026-02-20T17:13:18.583 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:13:18.583 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:18.583 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.590 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:13:18.590 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:18.590 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.597 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:13:18.597 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_sleep = '' " 2026-02-20T17:13:18.597 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:18.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 8 2026-02-20T17:13:26.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-20T17:13:09.905635+0000 2026-02-20T17:13:26.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2026-02-20T17:13:26.675 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:26.675 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:13:26.676 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:13:26.676 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T17:13:26.675162+0000" 2026-02-20T17:13:26.676 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:26.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:621: TEST_dump_scrub_schedule: sleep 1 2026-02-20T17:13:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:622: TEST_dump_scrub_schedule: sched_data=() 2026-02-20T17:13:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2026-02-20T17:13:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2026-02-20T17:13:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:624: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-20T17:13:09.905635+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2026-02-20T17:13:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T17:13:09.905635+0000 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:13:27.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:13:27.687 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2026-02-20T17:13:09.905635+0000 (10 retries) 2026-02-20T17:13:28.338 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-20T17:13:27.747610+0000", 2026-02-20T17:13:28.350 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:13:28.351 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:28.351 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 18, 2026-02-20T17:13:28.351 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 134 2026-02-20T17:13:28.351 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:28.351 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:28.423 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:28.423 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:13:28.423 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:13:28.423 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:13:28.423 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:13:28.424 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-20T17:13:27.747610+0000", 2026-02-20T17:13:28.424 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "queued for scrub at 2026-02-19T17:13:26.675" 2026-02-20T17:13:28.424 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:28.424 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=18 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=135 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='queued for scrub at 2026-02-19T17:13:26.675' 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='0' 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='0' 2026-02-20T17:13:28.507 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-18T17:13:26.675162+0000' 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=false 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:28.508 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-20T17:13:27.747610+0000' 2026-02-20T17:13:28.509 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:13:28.509 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:28.509 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=18 2026-02-20T17:13:28.509 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=134 2026-02-20T17:13:28.509 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:28.517 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ false / 135 / 134 / false / 2026-02-18T17:13:26.675162+0000 / queued for scrub at 2026-02-19T17:13:26.675 %%% query_is_future 2026-02-20T17:13:28.517 INFO:tasks.workunit.client.0.trial013.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2026-02-20T17:13:28.517 INFO:tasks.workunit.client.0.trial013.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2026-02-20T17:13:28.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:13:28.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-18T17:13:26.675162+0000 2026-02-20T17:13:28.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: ceph osd unset noscrub 2026-02-20T17:13:28.722 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is unset 2026-02-20T17:13:28.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2026-02-20T17:13:28.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: declare -A cond_active 2026-02-20T17:13:28.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: sched_data=() 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-18T17:13:26.675162+0000 cond_active 'WaitingActive ' sched_data 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-18T17:13:26.675162+0000 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:13:28.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:13:28.737 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-18T17:13:26.675162+0000 (10 retries) 2026-02-20T17:13:29.386 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:29.396 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:29.396 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:29.396 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:29.396 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-20T17:13:27.747610+0000", 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": false, 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 18, 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 134 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:29.397 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:29.471 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-20T17:13:29.717237+0000", 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-20T17:13:29.717 (2026-02-19T17:13:26.675)" 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:29.472 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:29.558 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:29.558 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=19 2026-02-20T17:13:29.558 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=138 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-20T17:13:29.717' 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-19T17:13:26.675' 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-18T17:13:26.675162+0000' 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:13:29.559 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-20T17:13:27.747610+0000' 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=false 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=18 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=134 2026-02-20T17:13:29.560 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:29.568 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ false / 138 / 134 / false / 2026-02-18T17:13:26.675162+0000 / scrub scheduled %%% query_active 2026-02-20T17:13:29.569 INFO:tasks.workunit.client.0.trial013.stdout:key is query_active: negation:0 # expected: true # in actual: false 2026-02-20T17:13:30.216 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-20T17:13:29.717237+0000", 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:30.226 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 19, 2026-02-20T17:13:30.227 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 137 2026-02-20T17:13:30.227 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:30.227 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "active": true, 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "epoch_start": "15", 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "start": "1:00000000::::0", 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "end": "MIN", 2026-02-20T17:13:30.302 INFO:tasks.workunit.client.0.trial013.stdout: "max_end": "MIN", 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "subset_last_update": "0'0", 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "req_scrub": false, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "auto_repair": false, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "check_repair": false, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "deep_scrub_on_error": false, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "priority": 5, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "shallow_errors": 0, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "deep_errors": 0, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "fixed": 0, 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "waiting_on_whom": [], 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrubbing", 2026-02-20T17:13:30.303 INFO:tasks.workunit.client.0.trial013.stdout: "is_reserving_replicas": false 2026-02-20T17:13:30.304 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:30.304 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:30.384 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=19 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=141 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=true 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrubbing' 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='0' 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='0' 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-18T17:13:26.675162+0000' 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:30.385 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=false 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-20T17:13:29.717237+0000' 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=19 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=137 2026-02-20T17:13:30.386 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:30.397 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 2 ~ true / 141 / 137 / false / 2026-02-18T17:13:26.675162+0000 / scrubbing %%% query_active 2026-02-20T17:13:30.397 INFO:tasks.workunit.client.0.trial013.stdout:key is query_active: negation:0 # expected: true # in actual: true 2026-02-20T17:13:30.397 INFO:tasks.workunit.client.0.trial013.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2026-02-20T17:13:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:13:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2026-02-20T17:13:30.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_dump_scrub_schedule: sched_data=() 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:647: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-18T17:13:26.675162+0000 cond_active_dmp 'WaitingActive ' sched_data 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-18T17:13:26.675162+0000 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:13:30.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:13:30.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:13:30.400 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-18T17:13:26.675162+0000 (10 retries) 2026-02-20T17:13:31.052 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-20T17:13:29.717237+0000", 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": false, 2026-02-20T17:13:31.062 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:13:31.063 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 19, 2026-02-20T17:13:31.063 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 137 2026-02-20T17:13:31.063 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:31.063 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:31.132 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "active": true, 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "epoch_start": "15", 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "start": "1:00000000::::0", 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "end": "MIN", 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "max_end": "MIN", 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "subset_last_update": "0'0", 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "req_scrub": false, 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "auto_repair": false, 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "check_repair": false, 2026-02-20T17:13:31.133 INFO:tasks.workunit.client.0.trial013.stdout: "deep_scrub_on_error": false, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "priority": 5, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "shallow_errors": 0, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "deep_errors": 0, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "fixed": 0, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "waiting_on_whom": [], 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrubbing", 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "is_reserving_replicas": false 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:31.134 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=19 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=142 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=true 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrubbing' 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='0' 2026-02-20T17:13:31.217 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='0' 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-18T17:13:26.675162+0000' 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=false 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:13:31.218 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-20T17:13:29.717237+0000' 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=false 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=19 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=137 2026-02-20T17:13:31.219 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:31.228 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ true / 142 / 137 / false / 2026-02-18T17:13:26.675162+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2026-02-20T17:13:31.228 INFO:tasks.workunit.client.0.trial013.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2026-02-20T17:13:31.228 INFO:tasks.workunit.client.0.trial013.stdout:key is query_active: negation:0 # expected: false # in actual: true 2026-02-20T17:13:31.876 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean+scrubbing", 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": true, 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "scrubbing for 2s", 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "0", 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": false, 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": false, 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 19, 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 141 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:31.886 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "active": true, 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "epoch_start": "15", 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "start": "1:a0217245:::obj31:0", 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "end": "1:a0217245:::obj31:0", 2026-02-20T17:13:31.960 INFO:tasks.workunit.client.0.trial013.stdout: "max_end": "1:a0217245:::obj31:0", 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "subset_last_update": "17'90", 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "req_scrub": false, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "auto_repair": false, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "check_repair": false, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "deep_scrub_on_error": false, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "priority": 5, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "shallow_errors": 0, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "deep_errors": 0, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "fixed": 0, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "waiting_on_whom": [], 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrubbing", 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "is_reserving_replicas": false 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:31.961 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:13:32.043 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:13:32.043 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=19 2026-02-20T17:13:32.043 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=143 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=true 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrubbing' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='0' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='0' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-18T17:13:26.675162+0000' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='17x90' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=false 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=null 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean+scrubbing' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=true 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='scrubbing for 2s' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='0' 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=false 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=false 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=19 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=141 2026-02-20T17:13:32.044 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:13:32.055 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 2 ~ true / 143 / 141 / false / 2026-02-18T17:13:26.675162+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2026-02-20T17:13:32.055 INFO:tasks.workunit.client.0.trial013.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: true 2026-02-20T17:13:32.055 INFO:tasks.workunit.client.0.trial013.stdout:WaitingActive - 'dmp_state_has_scrubbing' actual value (true) matches expected (true) (negation: 0) 2026-02-20T17:13:32.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:13:32.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:648: TEST_dump_scrub_schedule: sleep 4 2026-02-20T17:13:36.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:649: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2026-02-20T17:13:36.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:13:36.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:13:36.058 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:13:36.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:13:36.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:13:36.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:13:36.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:13:36.131 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:36.131 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.132 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.133 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.134 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.135 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.136 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.137 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:13:36.137 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:36.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:13:36.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:13:36.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:13:36.201 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:36.201 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:13:36.201 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.201 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.202 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 7, 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.203 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.204 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.205 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 2, 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:13:36.206 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:36.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:13:36.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:13:36.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.272 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:13:36.273 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:13:36.274 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.275 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:13:36.276 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:13:36.277 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.277 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:13:36.277 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:13:36.277 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:36.277 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2026-02-20T17:13:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2026-02-20T17:13:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:13:36.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:13:36.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:13:36.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:13:36.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:13:36.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:13:36.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:13:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:13:36.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:13:36.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:13:36.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:13:36.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:13:36.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:13:36.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:13:36.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:13:36.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:13:36.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:13:36.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:13:36.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.401 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:13:36.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:13:36.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:13:36.403 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2026-02-20T17:13:36.403 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2026-02-20T17:13:36.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:13:36.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2026-02-20T17:13:36.404 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:13:36.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:13:36.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:13:36.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:13:36.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:13:36.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:13:36.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:13:36.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:13:36.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:13:36.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:13:36.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:13:36.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:13:36.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:13:36.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:13:36.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:13:36.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:13:36.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:13:36.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:13:36.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:13:36.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:13:36.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:13:36.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:13:36.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:13:36.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:13:36.419 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:13:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:13:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:13:36.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:13:36.421 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2026-02-20T17:13:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2026-02-20T17:13:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2026-02-20T17:13:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: TEST_interval_changes: local poolname=test 2026-02-20T17:13:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:143: TEST_interval_changes: local OSDS=2 2026-02-20T17:13:36.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: local objects=10 2026-02-20T17:13:36.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: expr 24 '*' 60 '*' 60 2026-02-20T17:13:36.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: local day=86400 2026-02-20T17:13:36.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: expr 86400 '*' 7 2026-02-20T17:13:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: local week=604800 2026-02-20T17:13:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:148: TEST_interval_changes: local min_interval=86400 2026-02-20T17:13:36.424 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local max_interval=604800 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: TESTDATA=testdata.515468 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:13:36.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2026-02-20T17:13:36.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:13:36.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:13:36.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:13:36.445 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:13:36.445 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.445 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:36.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-20T17:13:36.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:13:36.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:13:36.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:13:36.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:13:36.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.477 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.478 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:13:36.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:13:36.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:13:36.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:13:36.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:13:36.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:13:36.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:13:36.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:13:36.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:13:36.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:36.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:13:36.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:13:36.704 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: expr 2 - 1 2026-02-20T17:13:36.705 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: seq 0 1 2026-02-20T17:13:36.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:13:36.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:13:36.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:36.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:13:36.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:13:36.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83cc61ae-ba23-49e3-8ff0-0afb12714f78 2026-02-20T17:13:36.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 83cc61ae-ba23-49e3-8ff0-0afb12714f78' 2026-02-20T17:13:36.714 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 83cc61ae-ba23-49e3-8ff0-0afb12714f78 2026-02-20T17:13:36.714 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:13:36.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAlphpXezhKhAA+C6oRaJPv1hwNBuSo3Q9qQ== 2026-02-20T17:13:36.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAlphpXezhKhAA+C6oRaJPv1hwNBuSo3Q9qQ=="}' 2026-02-20T17:13:36.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83cc61ae-ba23-49e3-8ff0-0afb12714f78 -i td/osd-scrub-test/0/new.json 2026-02-20T17:13:36.809 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:13:36.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:13:36.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBAlphpXezhKhAA+C6oRaJPv1hwNBuSo3Q9qQ== --osd-uuid 83cc61ae-ba23-49e3-8ff0-0afb12714f78 2026-02-20T17:13:36.834 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:36.829+0000 7f0f930b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:36.836 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:36.829+0000 7f0f930b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:36.837 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:36.833+0000 7f0f930b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:36.837 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:36.833+0000 7f0f930b7a40 -1 bdev(0x55aba87f4c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:13:36.837 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:36.833+0000 7f0f930b7a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:13:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:13:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:13:37.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:13:37.604 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:13:37.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:13:37.706 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:13:37.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:13:37.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-20T17:13:37.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:13:37.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:13:37.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:13:37.731 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:37.725+0000 7f48d9553a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:37.733 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:37.729+0000 7f48d9553a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:37.734 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:37.729+0000 7f48d9553a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:13:37.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:37.802 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:13:37.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:13:37.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:37.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:13:37.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:13:38.154 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:38.149+0000 7f48d9553a40 -1 Falling back to public interface 2026-02-20T17:13:38.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:13:38.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:38.895 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:13:38.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:13:38.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:38.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:13:38.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:13:39.065 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:39.061+0000 7f48d9553a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:13:39.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:13:39.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:39.988 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:13:39.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:13:39.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:39.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:13:40.147 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3461503316,v1:127.0.0.1:6801/3461503316] [v2:127.0.0.1:6802/3461503316,v1:127.0.0.1:6803/3461503316] exists,up 83cc61ae-ba23-49e3-8ff0-0afb12714f78 2026-02-20T17:13:40.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:13:40.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:13:40.152 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:13:40.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:13:40.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:13:40.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=63081d73-1695-4a10-a5d8-d943542f13fc 2026-02-20T17:13:40.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 63081d73-1695-4a10-a5d8-d943542f13fc' 2026-02-20T17:13:40.156 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 63081d73-1695-4a10-a5d8-d943542f13fc 2026-02-20T17:13:40.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:13:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBElphpywCtCRAA+aWv3aBCd4Mghrp9aEWlJw== 2026-02-20T17:13:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBElphpywCtCRAA+aWv3aBCd4Mghrp9aEWlJw=="}' 2026-02-20T17:13:40.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 63081d73-1695-4a10-a5d8-d943542f13fc -i td/osd-scrub-test/1/new.json 2026-02-20T17:13:40.316 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:13:40.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:13:40.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBElphpywCtCRAA+aWv3aBCd4Mghrp9aEWlJw== --osd-uuid 63081d73-1695-4a10-a5d8-d943542f13fc 2026-02-20T17:13:40.344 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:40.341+0000 7f2228370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:40.345 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:40.341+0000 7f2228370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:40.346 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:40.341+0000 7f2228370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:40.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:40.341+0000 7f2228370a40 -1 bdev(0x55aa7b9e6c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:13:40.347 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:40.341+0000 7f2228370a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:13:41.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:13:41.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:13:41.044 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:13:41.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:13:41.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:13:41.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:13:41.242 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:13:41.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-20T17:13:41.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:13:41.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:13:41.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:13:41.266 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:41.261+0000 7f88f8176a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:41.268 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:41.265+0000 7f88f8176a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:41.269 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:41.265+0000 7f88f8176a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:13:41.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:41.401 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:13:41.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:13:41.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:41.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:13:41.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:13:41.662 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:41.657+0000 7f88f8176a40 -1 Falling back to public interface 2026-02-20T17:13:42.308 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:13:42.301+0000 7f88f8176a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:13:42.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:13:42.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:42.559 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:13:42.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:13:42.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:42.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:13:42.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:13:43.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:13:43.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:13:43.719 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:13:43.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:13:43.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:13:43.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:13:43.875 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2827441841,v1:127.0.0.1:6811/2827441841] [v2:127.0.0.1:6812/2827441841,v1:127.0.0.1:6813/2827441841] exists,up 63081d73-1695-4a10-a5d8-d943542f13fc 2026-02-20T17:13:43.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:13:43.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:13:43.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:13:43.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: create_pool test 1 1 2026-02-20T17:13:43.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:13:44.078 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:13:44.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:13:45.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: wait_for_clean 2026-02-20T17:13:45.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:13:45.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:13:45.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:13:45.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:13:45.092 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:13:45.092 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:13:45.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:13:45.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:13:45.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:13:45.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:13:45.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:13:45.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:13:45.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:13:45.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:13:45.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:13:45.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:13:45.365 INFO:tasks.workunit.client.0.trial013.stderr:1' 2026-02-20T17:13:45.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:13:45.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:13:45.366 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:13:45.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-20T17:13:45.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-20T17:13:45.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-20T17:13:45.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:13:45.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:13:45.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-20T17:13:45.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-20T17:13:45.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2026-02-20T17:13:45.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:13:45.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-20T17:13:45.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:13:45.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:13:45.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-20T17:13:45.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:13:45.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-20T17:13:45.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-20T17:13:45.514 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836487 2026-02-20T17:13:45.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:13:45.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-20T17:13:45.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:13:46.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:13:46.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:13:46.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-20T17:13:46.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:13:47.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:13:47.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:13:48.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-20T17:13:48.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:13:48.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-20T17:13:48.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:13:48.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:13:48.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-20T17:13:48.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:13:48.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-20T17:13:48.010 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672964 2026-02-20T17:13:48.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-20T17:13:48.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:13:48.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672964 2026-02-20T17:13:48.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:13:48.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:13:48.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:13:48.377 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:13:48.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:13:48.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:13:48.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:13:48.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:13:48.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:13:48.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:13:48.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:13:48.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:13:48.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:13:48.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: ceph osd dump 2026-02-20T17:13:48.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2026-02-20T17:13:48.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: awk '{ print $2 }' 2026-02-20T17:13:48.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: local poolid=1 2026-02-20T17:13:48.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:13:48.888 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:13:48.888 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:13:48.888 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5897e-05 s, 28.7 MB/s 2026-02-20T17:13:48.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: seq 1 10 2026-02-20T17:13:48.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj1 testdata.515468 2026-02-20T17:13:48.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj2 testdata.515468 2026-02-20T17:13:48.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj3 testdata.515468 2026-02-20T17:13:48.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj4 testdata.515468 2026-02-20T17:13:48.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.970 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj5 testdata.515468 2026-02-20T17:13:48.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:48.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj6 testdata.515468 2026-02-20T17:13:49.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:49.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj7 testdata.515468 2026-02-20T17:13:49.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:49.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj8 testdata.515468 2026-02-20T17:13:49.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:49.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj9 testdata.515468 2026-02-20T17:13:49.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-20T17:13:49.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj10 testdata.515468 2026-02-20T17:13:49.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: rm -f testdata.515468 2026-02-20T17:13:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: get_primary test obj1 2026-02-20T17:13:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:13:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:13:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:13:49.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:13:49.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: local primary=1 2026-02-20T17:13:49.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: check_dump_scrubs 1 '1 day' 2026-02-20T17:13:49.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-20T17:13:49.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=1 day' 2026-02-20T17:13:49.282 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:49.283 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:13:49.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-20T17:13:49.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok dump_scrubs 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-21T17:13:44.076860+0000", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-21T17:13:44.076860+0000", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "level": "shallow", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: }, 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-28T06:51:53.132945+0000", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-28T06:51:53.132945+0000", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "level": "deep", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:13:49.343 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:13:49.344 INFO:tasks.workunit.client.0.trial013.stderr: } 2026-02-20T17:13:49.344 INFO:tasks.workunit.client.0.trial013.stderr:]' 2026-02-20T17:13:49.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-21T17:13:44.076860+0000",' '"orig_sched_time":' '"2026-02-21T17:13:44.076860+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-28T06:51:53.132945+0000",' '"orig_sched_time":' '"2026-02-28T06:51:53.132945+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-20T17:13:49.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-20T17:13:49.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-21T17:13:44.076860+0000"' 2026-02-20T17:13:49.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-21T17:13:44.076860+0000 2026-02-20T17:13:49.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-21T17:13:44.076860+0000 2026-02-20T17:13:49.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-20T17:13:49.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2026-02-20T17:13:49.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-21 = 2026-02-21 2026-02-20T17:13:49.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: get_asok_path osd.1 2026-02-20T17:13:49.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:13:49.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:13:49.358 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:13:49.358 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:13:49.358 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:13:49.358 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:13:49.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: expr 86400 '*' 2 2026-02-20T17:13:49.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: CEPH_ARGS= 2026-02-20T17:13:49.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2026-02-20T17:13:49.413 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:13:49.413 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-20T17:13:49.413 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:13:49.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: sleep 15 2026-02-20T17:14:04.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:182: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-20T17:14:04.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-20T17:14:04.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-20T17:14:04.422 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-20T17:14:04.422 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:14:04.422 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:14:04.423 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:14:04.423 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:04.423 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:04.423 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:14:04.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-20T17:14:04.424 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok dump_scrubs 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-22T17:13:44.076860+0000", 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-22T17:13:44.076860+0000", 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "level": "shallow", 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:04.483 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: }, 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-28T17:11:06.274140+0000", 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-28T17:11:06.274140+0000", 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "level": "deep", 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr: } 2026-02-20T17:14:04.484 INFO:tasks.workunit.client.0.trial013.stderr:]' 2026-02-20T17:14:04.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-20T17:14:04.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-22T17:13:44.076860+0000",' '"orig_sched_time":' '"2026-02-22T17:13:44.076860+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-28T17:11:06.274140+0000",' '"orig_sched_time":' '"2026-02-28T17:11:06.274140+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-20T17:14:04.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-22T17:13:44.076860+0000"' 2026-02-20T17:14:04.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-22T17:13:44.076860+0000 2026-02-20T17:14:04.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-22T17:13:44.076860+0000 2026-02-20T17:14:04.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-20T17:14:04.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-20T17:14:04.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-22 = 2026-02-22 2026-02-20T17:14:04.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: get_asok_path osd.1 2026-02-20T17:14:04.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:14:04.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:14:04.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:14:04.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:04.499 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:04.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:14:04.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: expr 604800 '*' 2 2026-02-20T17:14:04.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: CEPH_ARGS= 2026-02-20T17:14:04.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2026-02-20T17:14:04.557 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:14:04.557 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_max_interval = '' " 2026-02-20T17:14:04.557 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:14:04.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:186: TEST_interval_changes: sleep 15 2026-02-20T17:14:19.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-20T17:14:19.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-20T17:14:19.565 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-20T17:14:19.566 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-20T17:14:19.566 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:14:19.566 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:14:19.567 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:14:19.567 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:19.567 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:19.567 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:14:19.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-20T17:14:19.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok dump_scrubs 2026-02-20T17:14:19.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-22T17:13:44.076860+0000", 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-22T17:13:44.076860+0000", 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "level": "shallow", 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:19.627 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: }, 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-28T15:44:37.479993+0000", 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-28T15:44:37.479993+0000", 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "level": "deep", 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:19.628 INFO:tasks.workunit.client.0.trial013.stderr: } 2026-02-20T17:14:19.629 INFO:tasks.workunit.client.0.trial013.stderr:]' 2026-02-20T17:14:19.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-22T17:13:44.076860+0000",' '"orig_sched_time":' '"2026-02-22T17:13:44.076860+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-28T15:44:37.479993+0000",' '"orig_sched_time":' '"2026-02-28T15:44:37.479993+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-20T17:14:19.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-20T17:14:19.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-22T17:13:44.076860+0000"' 2026-02-20T17:14:19.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-22T17:13:44.076860+0000 2026-02-20T17:14:19.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-22T17:13:44.076860+0000 2026-02-20T17:14:19.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-20T17:14:19.640 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-20T17:14:19.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-22 = 2026-02-22 2026-02-20T17:14:19.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: expr 86400 '*' 3 2026-02-20T17:14:19.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2026-02-20T17:14:19.862 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 scrub_min_interval to 259200 2026-02-20T17:14:19.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:191: TEST_interval_changes: sleep 15 2026-02-20T17:14:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-20T17:14:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-20T17:14:34.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-20T17:14:34.877 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-20T17:14:34.877 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:14:34.877 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:14:34.878 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:14:34.878 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:34.878 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:34.879 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:14:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-20T17:14:34.879 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok dump_scrubs 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-23T17:13:44.076860+0000", 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-23T17:13:44.076860+0000", 2026-02-20T17:14:34.938 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "level": "shallow", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: }, 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-26T08:42:51.143336+0000", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-26T08:42:51.143336+0000", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "level": "deep", 2026-02-20T17:14:34.939 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:34.940 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:34.940 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:34.940 INFO:tasks.workunit.client.0.trial013.stderr: } 2026-02-20T17:14:34.940 INFO:tasks.workunit.client.0.trial013.stderr:]' 2026-02-20T17:14:34.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-20T17:14:34.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-23T17:13:44.076860+0000",' '"orig_sched_time":' '"2026-02-23T17:13:44.076860+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-26T08:42:51.143336+0000",' '"orig_sched_time":' '"2026-02-26T08:42:51.143336+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-20T17:14:34.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-23T17:13:44.076860+0000"' 2026-02-20T17:14:34.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-23T17:13:44.076860+0000 2026-02-20T17:14:34.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-23T17:13:44.076860+0000 2026-02-20T17:14:34.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-20T17:14:34.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-20T17:14:34.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-23 = 2026-02-23 2026-02-20T17:14:34.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: expr 604800 '*' 3 2026-02-20T17:14:34.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2026-02-20T17:14:35.154 INFO:tasks.workunit.client.0.trial013.stderr:set pool 1 scrub_max_interval to 1814400 2026-02-20T17:14:35.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:196: TEST_interval_changes: sleep 15 2026-02-20T17:14:50.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-20T17:14:50.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-20T17:14:50.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-20T17:14:50.167 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-20T17:14:50.168 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-20T17:14:50.168 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-20T17:14:50.168 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:14:50.169 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:50.169 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:50.169 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.1.asok 2026-02-20T17:14:50.169 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-20T17:14:50.170 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.515468/ceph-osd.1.asok dump_scrubs 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-02-23T17:13:44.076860+0000", 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-02-23T17:13:44.076860+0000", 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:50.230 INFO:tasks.workunit.client.0.trial013.stderr: "level": "shallow", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: }, 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: { 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "pgid": "1.0", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "sched_time": "2026-03-01T16:12:00.254858+0000", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "orig_sched_time": "2026-03-01T16:12:00.254858+0000", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "forced": false, 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "level": "deep", 2026-02-20T17:14:50.231 INFO:tasks.workunit.client.0.trial013.stderr: "urgency": "periodic-regular", 2026-02-20T17:14:50.232 INFO:tasks.workunit.client.0.trial013.stderr: "eligible": false, 2026-02-20T17:14:50.232 INFO:tasks.workunit.client.0.trial013.stderr: "last_issue": "ok" 2026-02-20T17:14:50.232 INFO:tasks.workunit.client.0.trial013.stderr: } 2026-02-20T17:14:50.232 INFO:tasks.workunit.client.0.trial013.stderr:]' 2026-02-20T17:14:50.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-20T17:14:50.233 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-23T17:13:44.076860+0000",' '"orig_sched_time":' '"2026-02-23T17:13:44.076860+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-01T16:12:00.254858+0000",' '"orig_sched_time":' '"2026-03-01T16:12:00.254858+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-20T17:14:50.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-23T17:13:44.076860+0000"' 2026-02-20T17:14:50.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-23T17:13:44.076860+0000 2026-02-20T17:14:50.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-23T17:13:44.076860+0000 2026-02-20T17:14:50.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-20T17:14:50.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-20T17:14:50.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-23 = 2026-02-23 2026-02-20T17:14:50.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2026-02-20T17:14:50.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:14:50.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2026-02-20T17:14:50.245 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2026-02-20T17:14:50.246 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2026-02-20T17:14:50.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:14:50.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:14:50.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:14:50.330 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:14:50.331 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.332 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.333 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.334 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:14:50.335 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:14:50.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:14:50.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:14:50.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:14:50.405 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:14:50.405 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:14:50.405 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.405 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.405 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.406 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.407 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.408 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:14:50.409 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:14:50.410 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:14:50.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2026-02-20T17:14:50.411 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2026-02-20T17:14:50.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:14:50.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:14:50.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:14:50.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:14:50.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:14:50.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:14:50.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:14:50.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:14:50.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:14:50.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:14:50.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:14:50.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:14:50.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:14:50.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:14:50.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:14:50.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:14:50.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:14:50.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:14:50.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:14:50.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:14:50.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:14:50.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:14:50.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:14:50.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:14:50.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:50.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:50.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:14:50.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:14:50.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:14:50.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2026-02-20T17:14:50.534 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2026-02-20T17:14:50.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:14:50.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2026-02-20T17:14:50.535 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:14:50.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:14:50.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:14:50.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:14:50.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:14:50.538 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:14:50.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:14:50.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:14:50.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:14:50.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:14:50.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:14:50.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:14:50.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:14:50.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:14:50.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:14:50.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:14:50.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:14:50.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:14:50.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:14:50.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:50.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:50.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:14:50.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:14:50.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:14:50.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:14:50.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:14:50.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:14:50.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:14:50.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:14:50.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:14:50.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:14:50.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:14:50.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:14:50.550 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2026-02-20T17:14:50.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2026-02-20T17:14:50.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2026-02-20T17:14:50.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:462: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: local -A cluster_conf 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:469: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-20T17:14:50.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-20T17:14:50.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-20T17:14:50.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-20T17:14:50.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-20T17:14:50.835 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 c29a9ea2-ef6d-4431-b241-ca6b6c511e26 2026-02-20T17:14:50.935 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:14:50.966 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:50.960+0000 7fdf0e503a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:50.967 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:50.960+0000 7fdf0e503a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:50.968 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:50.964+0000 7fdf0e503a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:50.969 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:50.964+0000 7fdf0e503a40 -1 bdev(0x55e0761bac00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:14:50.969 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:50.964+0000 7fdf0e503a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:14:51.729 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:14:51.835 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:14:51.866 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:51.860+0000 7f66edbb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:51.868 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:51.864+0000 7f66edbb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:51.869 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:51.864+0000 7f66edbb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:51.928 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:14:52.294 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:52.288+0000 7f66edbb0a40 -1 Falling back to public interface 2026-02-20T17:14:53.021 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:14:53.279 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:53.272+0000 7f66edbb0a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:14:54.114 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:14:54.274 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3671155704,v1:127.0.0.1:6801/3671155704] [v2:127.0.0.1:6802/3671155704,v1:127.0.0.1:6803/3671155704] exists,up c29a9ea2-ef6d-4431-b241-ca6b6c511e26 2026-02-20T17:14:54.279 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 9bfe4f19-c2e7-4ef1-a80e-c6975e482c3e 2026-02-20T17:14:54.454 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:14:54.495 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:54.488+0000 7f4b67cbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:54.497 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:54.492+0000 7f4b67cbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:54.498 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:54.492+0000 7f4b67cbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:54.499 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:54.492+0000 7f4b67cbea40 -1 bdev(0x563e8c5c2c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:14:54.499 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:54.492+0000 7f4b67cbea40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:14:55.127 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:14:55.329 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:14:55.357 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:55.352+0000 7f7715952a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:55.359 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:55.352+0000 7f7715952a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:55.360 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:55.356+0000 7f7715952a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:55.488 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:14:55.782 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:55.780+0000 7f7715952a40 -1 Falling back to public interface 2026-02-20T17:14:56.378 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:56.376+0000 7f7715952a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:14:56.645 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:14:57.799 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:14:57.953 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2776631450,v1:127.0.0.1:6811/2776631450] [v2:127.0.0.1:6812/2776631450,v1:127.0.0.1:6813/2776631450] exists,up 9bfe4f19-c2e7-4ef1-a80e-c6975e482c3e 2026-02-20T17:14:57.958 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 678a0392-f003-44b9-8bad-05feac1ff23e 2026-02-20T17:14:58.133 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:14:58.164 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:58.160+0000 7f04acacea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:58.165 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:58.164+0000 7f04acacea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:58.166 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:58.164+0000 7f04acacea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:58.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:58.164+0000 7f04acacea40 -1 bdev(0x55c4fb942c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:14:58.167 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:58.164+0000 7f04acacea40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:14:58.815 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:14:59.022 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:14:59.051 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:59.048+0000 7f3dc6a4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:59.053 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:59.052+0000 7f3dc6a4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:59.054 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:59.052+0000 7f3dc6a4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:14:59.175 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:14:59.414 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:14:59.412+0000 7f3dc6a4ca40 -1 Falling back to public interface 2026-02-20T17:15:00.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:00.052+0000 7f3dc6a4ca40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:15:00.337 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:01.496 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:01.655 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1562539698,v1:127.0.0.1:6819/1562539698] [v2:127.0.0.1:6820/1562539698,v1:127.0.0.1:6821/1562539698] exists,up 678a0392-f003-44b9-8bad-05feac1ff23e 2026-02-20T17:15:01.861 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:15:03.365 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T17:15:05.851 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672967 2026-02-20T17:15:06.022 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 64424509444 2026-02-20T17:15:07.881 INFO:tasks.workunit.client.0.trial013.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-20T17:15:07.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:470: TEST_just_deep_scrubs: local poolid=1 2026-02-20T17:15:07.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:471: TEST_just_deep_scrubs: local poolname=test 2026-02-20T17:15:07.882 INFO:tasks.workunit.client.0.trial013.stdout:Pool: test : 1 2026-02-20T17:15:07.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:472: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2026-02-20T17:15:07.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: TESTDATA=testdata.515468 2026-02-20T17:15:07.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local objects=90 2026-02-20T17:15:07.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:15:07.883 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:15:07.884 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:15:07.884 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.9151e-05 s, 17.4 MB/s 2026-02-20T17:15:07.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: seq 1 90 2026-02-20T17:15:07.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj1 testdata.515468 2026-02-20T17:15:07.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj2 testdata.515468 2026-02-20T17:15:07.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj3 testdata.515468 2026-02-20T17:15:07.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj4 testdata.515468 2026-02-20T17:15:07.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj5 testdata.515468 2026-02-20T17:15:07.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:07.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj6 testdata.515468 2026-02-20T17:15:08.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj7 testdata.515468 2026-02-20T17:15:08.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj8 testdata.515468 2026-02-20T17:15:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj9 testdata.515468 2026-02-20T17:15:08.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj10 testdata.515468 2026-02-20T17:15:08.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj11 testdata.515468 2026-02-20T17:15:08.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj12 testdata.515468 2026-02-20T17:15:08.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj13 testdata.515468 2026-02-20T17:15:08.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj14 testdata.515468 2026-02-20T17:15:08.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj15 testdata.515468 2026-02-20T17:15:08.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj16 testdata.515468 2026-02-20T17:15:08.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj17 testdata.515468 2026-02-20T17:15:08.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj18 testdata.515468 2026-02-20T17:15:08.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj19 testdata.515468 2026-02-20T17:15:08.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj20 testdata.515468 2026-02-20T17:15:08.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj21 testdata.515468 2026-02-20T17:15:08.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj22 testdata.515468 2026-02-20T17:15:08.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj23 testdata.515468 2026-02-20T17:15:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj24 testdata.515468 2026-02-20T17:15:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj25 testdata.515468 2026-02-20T17:15:08.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.472 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj26 testdata.515468 2026-02-20T17:15:08.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj27 testdata.515468 2026-02-20T17:15:08.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj28 testdata.515468 2026-02-20T17:15:08.536 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj29 testdata.515468 2026-02-20T17:15:08.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj30 testdata.515468 2026-02-20T17:15:08.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj31 testdata.515468 2026-02-20T17:15:08.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.612 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj32 testdata.515468 2026-02-20T17:15:08.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj33 testdata.515468 2026-02-20T17:15:08.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj34 testdata.515468 2026-02-20T17:15:08.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj35 testdata.515468 2026-02-20T17:15:08.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj36 testdata.515468 2026-02-20T17:15:08.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj37 testdata.515468 2026-02-20T17:15:08.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj38 testdata.515468 2026-02-20T17:15:08.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj39 testdata.515468 2026-02-20T17:15:08.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj40 testdata.515468 2026-02-20T17:15:08.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj41 testdata.515468 2026-02-20T17:15:08.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj42 testdata.515468 2026-02-20T17:15:08.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj43 testdata.515468 2026-02-20T17:15:08.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj44 testdata.515468 2026-02-20T17:15:08.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj45 testdata.515468 2026-02-20T17:15:08.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj46 testdata.515468 2026-02-20T17:15:08.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj47 testdata.515468 2026-02-20T17:15:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:08.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj48 testdata.515468 2026-02-20T17:15:09.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj49 testdata.515468 2026-02-20T17:15:09.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj50 testdata.515468 2026-02-20T17:15:09.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj51 testdata.515468 2026-02-20T17:15:09.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj52 testdata.515468 2026-02-20T17:15:09.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj53 testdata.515468 2026-02-20T17:15:09.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj54 testdata.515468 2026-02-20T17:15:09.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj55 testdata.515468 2026-02-20T17:15:09.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj56 testdata.515468 2026-02-20T17:15:09.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj57 testdata.515468 2026-02-20T17:15:09.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj58 testdata.515468 2026-02-20T17:15:09.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj59 testdata.515468 2026-02-20T17:15:09.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj60 testdata.515468 2026-02-20T17:15:09.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj61 testdata.515468 2026-02-20T17:15:09.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj62 testdata.515468 2026-02-20T17:15:09.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj63 testdata.515468 2026-02-20T17:15:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj64 testdata.515468 2026-02-20T17:15:09.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj65 testdata.515468 2026-02-20T17:15:09.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj66 testdata.515468 2026-02-20T17:15:09.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj67 testdata.515468 2026-02-20T17:15:09.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj68 testdata.515468 2026-02-20T17:15:09.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj69 testdata.515468 2026-02-20T17:15:09.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj70 testdata.515468 2026-02-20T17:15:09.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj71 testdata.515468 2026-02-20T17:15:09.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.583 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj72 testdata.515468 2026-02-20T17:15:09.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj73 testdata.515468 2026-02-20T17:15:09.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj74 testdata.515468 2026-02-20T17:15:09.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj75 testdata.515468 2026-02-20T17:15:09.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj76 testdata.515468 2026-02-20T17:15:09.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj77 testdata.515468 2026-02-20T17:15:09.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj78 testdata.515468 2026-02-20T17:15:09.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj79 testdata.515468 2026-02-20T17:15:09.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj80 testdata.515468 2026-02-20T17:15:09.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj81 testdata.515468 2026-02-20T17:15:09.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj82 testdata.515468 2026-02-20T17:15:09.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj83 testdata.515468 2026-02-20T17:15:09.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj84 testdata.515468 2026-02-20T17:15:09.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj85 testdata.515468 2026-02-20T17:15:09.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj86 testdata.515468 2026-02-20T17:15:09.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj87 testdata.515468 2026-02-20T17:15:09.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:09.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj88 testdata.515468 2026-02-20T17:15:10.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:10.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj89 testdata.515468 2026-02-20T17:15:10.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-20T17:15:10.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj90 testdata.515468 2026-02-20T17:15:10.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: rm -f testdata.515468 2026-02-20T17:15:10.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: ceph tell 'osd.*' config set osd_scrub_retry_after_noscrub 2 2026-02-20T17:15:10.102 INFO:tasks.workunit.client.0.trial013.stdout:osd.0: { 2026-02-20T17:15:10.102 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-20T17:15:10.103 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:10.109 INFO:tasks.workunit.client.0.trial013.stdout:osd.1: { 2026-02-20T17:15:10.110 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-20T17:15:10.110 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:10.116 INFO:tasks.workunit.client.0.trial013.stdout:osd.2: { 2026-02-20T17:15:10.117 INFO:tasks.workunit.client.0.trial013.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-20T17:15:10.117 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:10.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:487: TEST_just_deep_scrubs: ceph osd set noscrub 2026-02-20T17:15:10.323 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:15:10.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:488: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2026-02-20T17:15:10.528 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is set 2026-02-20T17:15:10.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:489: TEST_just_deep_scrubs: sleep 6 2026-02-20T17:15:16.541 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: date -Ins 2026-02-20T17:15:16.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: local now_is=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: declare -A sched_data 2026-02-20T17:15:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: local pgid=1.2 2026-02-20T17:15:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: set_query_debug 1.2 2026-02-20T17:15:16.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.2 2026-02-20T17:15:16.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-20T17:15:16.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-20T17:15:16.690 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs_brief 2026-02-20T17:15:16.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=0 2026-02-20T17:15:16.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2026-02-20T17:15:16.701 INFO:tasks.workunit.client.0.trial013.stdout:Setting scrub debug data. Primary for 1.2 is 0 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.0 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-osd.0.asok 2026-02-20T17:15:16.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-20T17:15:16.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.515468/ceph-osd.0.asok scrubdebug 1.2 set sessions 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:497: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16,541956501+00:00 sched_data 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-20T17:15:16.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-20T17:15:16.921 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout:{"success":true}from pg dump pg: { 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 0, 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:15:01.860645+0000", 2026-02-20T17:15:16.931 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:15:16.932 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:15:16.932 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 21, 2026-02-20T17:15:16.932 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 42 2026-02-20T17:15:16.932 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:16.932 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:15:01.860645+0000", 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:15:01.860 (2026-02-21T17:15:01.860)", 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: "test_sequence": 2342912 2026-02-20T17:15:17.003 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:17.004 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=21 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=43 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:15:01.860' 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:15:01.860' 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=0 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:15:01.860645+0000' 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='0x0' 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:15:17.087 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=2342912 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=0 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:15:01.860645+0000' 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=21 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=42 2026-02-20T17:15:17.088 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:15:17.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: local saved_last_stamp=2026-02-20T17:15:01.860645+0000 2026-02-20T17:15:17.100 INFO:tasks.workunit.client.0.trial013.stdout:test counter @ start: 2342912 2026-02-20T17:15:17.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: local dbg_counter_at_start=2342912 2026-02-20T17:15:17.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:500: TEST_just_deep_scrubs: echo 'test counter @ start: 2342912' 2026-02-20T17:15:17.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2026-02-20T17:15:17.164 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:17.164 INFO:tasks.workunit.client.0.trial013.stdout: "deep": true, 2026-02-20T17:15:17.164 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:15:17.164 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-09T12:27:07.163420+0000" 2026-02-20T17:15:17.164 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:17.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: sleep 5 2026-02-20T17:15:22.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-20T17:15:22.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: jq -r '.pg_stats[] | "\(.pgid) \(.stat_sum.num_objects)"' 2026-02-20T17:15:22.316 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:15:22.327 INFO:tasks.workunit.client.0.trial013.stdout:1.3 20 2026-02-20T17:15:22.327 INFO:tasks.workunit.client.0.trial013.stdout:1.2 20 2026-02-20T17:15:22.327 INFO:tasks.workunit.client.0.trial013.stdout:1.1 25 2026-02-20T17:15:22.327 INFO:tasks.workunit.client.0.trial013.stdout:1.0 25 2026-02-20T17:15:22.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: ceph pg 1.2 query --format=json-pretty 2026-02-20T17:15:22.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: jq -r .info.stats.stat_sum.num_objects 2026-02-20T17:15:22.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: echo 'Objects # in pg 1.2: ' 20 2026-02-20T17:15:22.399 INFO:tasks.workunit.client.0.trial013.stdout:Objects # in pg 1.2: 20 2026-02-20T17:15:22.400 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16,541956501+00:00 sc_data_2 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-20T17:15:22.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-20T17:15:22.545 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:15:22.557 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 0, 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic deep scrub scheduled", 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-20T17:15:19.303631+0000", 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 21, 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 47 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:22.558 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-20T17:15:24.283362+0000", 2026-02-20T17:15:22.628 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "deep scrub scheduled @ 2026-02-20T17:15:24.283 (2026-02-16T01:13:28.033)", 2026-02-20T17:15:22.629 INFO:tasks.workunit.client.0.trial013.stdout: "test_sequence": 2342912 2026-02-20T17:15:22.629 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:22.629 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=21 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=48 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='deep scrub scheduled' 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-20T17:15:24.283' 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-16T01:13:28.033' 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=0 2026-02-20T17:15:22.713 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:15:01.860645+0000' 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='0x0' 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=false 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=2342912 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=0 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-20T17:15:19.303631+0000' 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:15:22.714 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:15:22.715 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=21 2026-02-20T17:15:22.715 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=47 2026-02-20T17:15:22.715 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:15:22.724 INFO:tasks.workunit.client.0.trial013.stdout:test counter @ should show no change: 2342912 2026-02-20T17:15:22.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 2342912 2026-02-20T17:15:22.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2026-02-20T17:15:22.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 2342912 == 2342912 )) 2026-02-20T17:15:22.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2026-02-20T17:15:22.922 INFO:tasks.workunit.client.0.trial013.stderr:nodeep-scrub is unset 2026-02-20T17:15:22.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 8 2026-02-20T17:15:30.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:16,541956501+00:00 sc_data_2 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-20T17:15:16,541956501+00:00 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-20T17:15:30.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-20T17:15:30.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-20T17:15:31.082 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:15:31.093 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:15:31.093 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:15:31.093 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:15:31.093 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:15:24.324351+0000", 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 22, 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 60 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:31.094 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:15:31.167 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:15:31.167 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:15:31.167 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:15:31.167 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:15:31.167 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:15:31.168 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:15:24.324351+0000", 2026-02-20T17:15:31.168 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:15:24.324 (2026-02-21T17:15:24.324)", 2026-02-20T17:15:31.168 INFO:tasks.workunit.client.0.trial013.stdout: "test_sequence": 2342915 2026-02-20T17:15:31.168 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:31.168 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:15:31.250 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:15:31.250 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=22 2026-02-20T17:15:31.250 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=61 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:15:24.324' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:15:24.324' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:15:24.324351+0000' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='19x20' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=2342915 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:15:24.324351+0000' 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=22 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=60 2026-02-20T17:15:31.251 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:15:31.261 INFO:tasks.workunit.client.0.trial013.stdout:test counter @ should be higher than before the unset: 2342915 2026-02-20T17:15:31.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2342915 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_just_deep_scrubs: sc_data_2=() 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:525: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2026-02-20T17:15:01.860645+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.2 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T17:15:01.860645+0000 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sc_data_2 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-20T17:15:31.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-20T17:15:31.263 INFO:tasks.workunit.client.0.trial013.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2026-02-20T17:15:01.860645+0000 (10 retries) 2026-02-20T17:15:31.912 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:15:31.922 INFO:tasks.workunit.client.0.trial013.stdout:from pg dump pg: { 2026-02-20T17:15:31.922 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_pg_state": "active+clean", 2026-02-20T17:15:31.922 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_state_has_scrubbing": false, 2026-02-20T17:15:31.922 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_last_duration": 1, 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_schedule_at": "2026-02-21T17:15:24.324351+0000", 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_is_future": true, 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_vs_date": true, 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_reported_epoch": 22, 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout: "dmp_seq": 60 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:31.923 INFO:tasks.workunit.client.0.trial013.stdout:query output: 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "scrubber": { 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "active": false, 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "must_scrub": false, 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "must_deep_scrub": false, 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "must_repair": false, 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "scrub_reg_stamp": "2026-02-21T17:15:24.324351+0000", 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "schedule": "scrub scheduled @ 2026-02-21T17:15:24.324 (2026-02-21T17:15:24.324)", 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "test_sequence": 2342915 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:31.996 INFO:tasks.workunit.client.0.trial013.stdout: "agent_state": {} 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:( 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:[query_epoch]=22 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:[query_seq]=61 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:[query_active]=false 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule]='scrub scheduled' 2026-02-20T17:15:32.079 INFO:tasks.workunit.client.0.trial013.stdout:[query_schedule_at]='2026-02-21T17:15:24.324' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_target_at]='2026-02-21T17:15:24.324' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_duration]=1 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_stamp]='2026-02-20T17:15:24.324351+0000' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_last_scrub]='19x20' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_is_future]=true 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_vs_date]=true 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[query_scrub_seq]=2342915 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_pg_state]='active+clean' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_state_has_scrubbing]=false 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_last_duration]=1 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-20T17:15:32.080 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_schedule_at]='2026-02-21T17:15:24.324351+0000' 2026-02-20T17:15:32.081 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_is_future]=true 2026-02-20T17:15:32.081 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_vs_date]=true 2026-02-20T17:15:32.081 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_reported_epoch]=22 2026-02-20T17:15:32.081 INFO:tasks.workunit.client.0.trial013.stdout:[dmp_seq]=60 2026-02-20T17:15:32.081 INFO:tasks.workunit.client.0.trial013.stdout:) 2026-02-20T17:15:32.089 INFO:tasks.workunit.client.0.trial013.stdout:--> loop: 1 ~ false / 61 / 60 / true / 2026-02-20T17:15:24.324351+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-20T17:15:32.090 INFO:tasks.workunit.client.0.trial013.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-20T17:15:32.090 INFO:tasks.workunit.client.0.trial013.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-20T17:15:32.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-20T17:15:32.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_just_deep_scrubs: perf_counters td/osd-scrub-test 3 2026-02-20T17:15:32.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:15:32.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:15:32.091 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:15:32.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:15:32.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:32.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:15:32.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.162 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.163 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 2, 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.164 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.165 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.166 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.167 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.167 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.167 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:32.167 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:32.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:15:32.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:32.233 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:32.233 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:32.233 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.234 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.235 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.236 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.237 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:32.238 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:32.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:32.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:15:32.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.302 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.303 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.304 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:32.305 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.306 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:32.307 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:32.307 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:32.307 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_just_deep_scrubs ------------------ 2026-02-20T17:15:32.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_just_deep_scrubs ------------------' 2026-02-20T17:15:32.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:32.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:32.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:32.413 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:32.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:32.414 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:32.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:32.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:32.416 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:32.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:32.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:32.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:32.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:32.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:32.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:32.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:32.433 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_just_deep_scrubs ------------------ 2026-02-20T17:15:32.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_just_deep_scrubs ------------------' 2026-02-20T17:15:32.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:15:32.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_pg_dump_objects_scrubbed -------------------' 2026-02-20T17:15:32.434 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_pg_dump_objects_scrubbed ------------------- 2026-02-20T17:15:32.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:32.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:32.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:32.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:32.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:32.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:32.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:32.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:32.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:32.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:32.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:32.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:32.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:32.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:32.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:32.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:32.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:15:32.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:15:32.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:15:32.450 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:15:32.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:15:32.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:15:32.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:15:32.451 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_pg_dump_objects_scrubbed ----------------------- 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_pg_dump_objects_scrubbed -----------------------' 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_pg_dump_objects_scrubbed td/osd-scrub-test 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: local dir=td/osd-scrub-test 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:654: TEST_pg_dump_objects_scrubbed: local poolname=test 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:655: TEST_pg_dump_objects_scrubbed: local OSDS=3 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:656: TEST_pg_dump_objects_scrubbed: local objects=15 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:657: TEST_pg_dump_objects_scrubbed: local timeout=10 2026-02-20T17:15:32.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: TESTDATA=testdata.515468 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: setup td/osd-scrub-test 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:32.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:32.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:32.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:32.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:32.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:32.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:32.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:32.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:32.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:32.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:32.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:32.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:32.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:32.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:32.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:32.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:32.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:32.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:32.464 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:15:32.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:15:32.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:15:32.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:15:32.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:15:32.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:15:32.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:15:32.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:662: TEST_pg_dump_objects_scrubbed: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:15:32.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:15:32.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:15:32.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:32.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:32.490 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:32.490 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.490 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:32.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:15:32.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:15:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:15:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:15:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:15:32.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:15:32.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:15:32.524 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:15:32.524 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:15:32.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:15:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:15:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:15:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:15:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:15:32.581 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:15:32.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:15:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:663: TEST_pg_dump_objects_scrubbed: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:15:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:15:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:15:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:15:32.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:15:32.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:15:32.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:15:32.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:15:32.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:32.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:32.732 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:32.732 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.732 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:32.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:15:32.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:15:32.751 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: expr 3 - 1 2026-02-20T17:15:32.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: seq 0 2 2026-02-20T17:15:32.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 0 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:15:32.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:15:32.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:15:32.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:15:32.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ffdafe5c-ca0c-42cc-8a51-a555ea82ccca 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ffdafe5c-ca0c-42cc-8a51-a555ea82ccca' 2026-02-20T17:15:32.762 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 ffdafe5c-ca0c-42cc-8a51-a555ea82ccca 2026-02-20T17:15:32.763 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:15:32.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0lphpJJe8LRAAArqJXVXzRoqSFqT0PmjFDg== 2026-02-20T17:15:32.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0lphpJJe8LRAAArqJXVXzRoqSFqT0PmjFDg=="}' 2026-02-20T17:15:32.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ffdafe5c-ca0c-42cc-8a51-a555ea82ccca -i td/osd-scrub-test/0/new.json 2026-02-20T17:15:32.856 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:32.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:15:32.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0lphpJJe8LRAAArqJXVXzRoqSFqT0PmjFDg== --osd-uuid ffdafe5c-ca0c-42cc-8a51-a555ea82ccca 2026-02-20T17:15:32.880 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:32.879+0000 7efcc6bbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:32.882 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:32.879+0000 7efcc6bbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:32.882 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:32.879+0000 7efcc6bbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:32.883 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:32.879+0000 7efcc6bbfa40 -1 bdev(0x55cb18c0ec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:15:32.883 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:32.879+0000 7efcc6bbfa40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:15:33.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:15:33.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:15:33.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:15:33.637 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:15:33.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:15:33.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:15:33.740 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:15:33.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:15:33.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:15:33.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:15:33.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:15:33.764 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:33.763+0000 7f4492941a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:33.767 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:33.763+0000 7f4492941a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:33.768 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:33.763+0000 7f4492941a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:15:33.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:33.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:15:33.833 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:33.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:33.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:33.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:34.302 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:34.299+0000 7f4492941a40 -1 Falling back to public interface 2026-02-20T17:15:34.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:34.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:34.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:15:34.930 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:34.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:35.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:35.250 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:35.247+0000 7f4492941a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:15:36.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:36.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:36.022 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:36.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:15:36.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:36.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:36.182 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2640735721,v1:127.0.0.1:6801/2640735721] [v2:127.0.0.1:6802/2640735721,v1:127.0.0.1:6803/2640735721] exists,up ffdafe5c-ca0c-42cc-8a51-a555ea82ccca 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 1 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:15:36.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:36.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:15:36.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:15:36.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:15:36.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:15:36.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:15:36.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:15:36.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:15:36.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22731af8-9c91-4ce3-a8b3-d9f91472ff74 2026-02-20T17:15:36.190 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 22731af8-9c91-4ce3-a8b3-d9f91472ff74 2026-02-20T17:15:36.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 22731af8-9c91-4ce3-a8b3-d9f91472ff74' 2026-02-20T17:15:36.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:15:36.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4lphp60/KCxAA1wY/8RmX5b51DnxB+K3xeQ== 2026-02-20T17:15:36.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4lphp60/KCxAA1wY/8RmX5b51DnxB+K3xeQ=="}' 2026-02-20T17:15:36.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22731af8-9c91-4ce3-a8b3-d9f91472ff74 -i td/osd-scrub-test/1/new.json 2026-02-20T17:15:36.364 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:36.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:15:36.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4lphp60/KCxAA1wY/8RmX5b51DnxB+K3xeQ== --osd-uuid 22731af8-9c91-4ce3-a8b3-d9f91472ff74 2026-02-20T17:15:36.390 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:36.387+0000 7f66953f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:36.391 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:36.387+0000 7f66953f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:36.392 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:36.391+0000 7f66953f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:36.393 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:36.391+0000 7f66953f5a40 -1 bdev(0x560b6dd14c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:15:36.393 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:36.391+0000 7f66953f5a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:15:37.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:15:37.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:15:37.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:15:37.009 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:15:37.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:15:37.202 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:15:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:15:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:15:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:15:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:15:37.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:15:37.217 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:37.215+0000 7f5bc9239a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:37.218 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:37.215+0000 7f5bc9239a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:37.219 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:37.215+0000 7f5bc9239a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:37.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:15:37.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:15:37.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:15:37.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:15:37.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:15:37.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:37.361 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:37.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:15:37.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:37.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:15:37.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:37.602 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:37.599+0000 7f5bc9239a40 -1 Falling back to public interface 2026-02-20T17:15:38.254 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:38.251+0000 7f5bc9239a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:15:38.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:38.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:38.521 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:15:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:38.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:15:38.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:39.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:39.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:39.681 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:39.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:15:39.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:39.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:15:39.845 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1845608309,v1:127.0.0.1:6811/1845608309] [v2:127.0.0.1:6812/1845608309,v1:127.0.0.1:6813/1845608309] exists,up 22731af8-9c91-4ce3-a8b3-d9f91472ff74 2026-02-20T17:15:39.848 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 2 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:15:39.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:39.850 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:15:39.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:15:39.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:15:39.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:15:39.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:15:39.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=88ec8d9c-6591-490d-8fe1-f60758634ef7 2026-02-20T17:15:39.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 88ec8d9c-6591-490d-8fe1-f60758634ef7' 2026-02-20T17:15:39.853 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 88ec8d9c-6591-490d-8fe1-f60758634ef7 2026-02-20T17:15:39.853 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:15:39.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7lphpSRdSMxAAi81Ar3zj10szVGwtMd39ng== 2026-02-20T17:15:39.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7lphpSRdSMxAAi81Ar3zj10szVGwtMd39ng=="}' 2026-02-20T17:15:39.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 88ec8d9c-6591-490d-8fe1-f60758634ef7 -i td/osd-scrub-test/2/new.json 2026-02-20T17:15:40.023 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:40.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:15:40.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7lphpSRdSMxAAi81Ar3zj10szVGwtMd39ng== --osd-uuid 88ec8d9c-6591-490d-8fe1-f60758634ef7 2026-02-20T17:15:40.058 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.055+0000 7f6fe1414a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:40.059 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.055+0000 7f6fe1414a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:40.060 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.059+0000 7f6fe1414a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:40.061 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.059+0000 7f6fe1414a40 -1 bdev(0x55ee92aa2c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:15:40.061 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.059+0000 7f6fe1414a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:15:40.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:15:40.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:15:40.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:15:40.712 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:15:40.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:15:40.907 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:15:40.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:15:40.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:15:40.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:15:40.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:15:40.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:15:40.930 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.927+0000 7fac232aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:40.932 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.927+0000 7fac232aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:40.933 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:40.931+0000 7fac232aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:41.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:15:41.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:15:41.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:15:41.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:15:41.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:15:41.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:41.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:15:41.067 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:41.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:41.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:15:41.229 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:41.338 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:41.335+0000 7fac232aca40 -1 Falling back to public interface 2026-02-20T17:15:42.014 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:42.011+0000 7fac232aca40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:15:42.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:42.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:42.231 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:42.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:15:42.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:15:42.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:42.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:43.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:43.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:43.392 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:43.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:15:43.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:43.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:15:43.550 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1714686875,v1:127.0.0.1:6819/1714686875] [v2:127.0.0.1:6820/1714686875,v1:127.0.0.1:6821/1714686875] exists,up 88ec8d9c-6591-490d-8fe1-f60758634ef7 2026-02-20T17:15:43.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:15:43.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:15:43.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:15:43.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: create_pool test 1 1 2026-02-20T17:15:43.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:15:43.755 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:15:43.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:15:44.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:671: TEST_pg_dump_objects_scrubbed: wait_for_clean 2026-02-20T17:15:44.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:15:44.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:15:44.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:15:44.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:15:44.768 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:15:44.768 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:15:44.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:15:44.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:15:44.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:15:44.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:15:44.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:15:44.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:15:44.877 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:15:44.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:15:44.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:45.032 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:15:45.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-20T17:15:45.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-20T17:15:45.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-20T17:15:45.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:45.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:15:45.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-20T17:15:45.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-20T17:15:45.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2026-02-20T17:15:45.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:45.175 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:15:45.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T17:15:45.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T17:15:45.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-60129542148' 2026-02-20T17:15:45.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-20T17:15:45.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:45.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:15:45.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-20T17:15:45.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:45.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-20T17:15:45.251 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T17:15:45.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-20T17:15:45.252 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:15:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T17:15:45.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:15:46.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:15:46.406 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:15:46.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T17:15:46.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:15:47.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:15:47.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:15:47.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836490 2026-02-20T17:15:47.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:47.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-20T17:15:47.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:47.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:15:47.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-20T17:15:47.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:47.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-20T17:15:47.734 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672967 2026-02-20T17:15:47.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-20T17:15:47.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:15:47.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672967 2026-02-20T17:15:47.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:47.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T17:15:47.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:47.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:15:47.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T17:15:47.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:47.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T17:15:47.909 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T17:15:47.909 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T17:15:47.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:15:48.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542148 2026-02-20T17:15:48.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:15:48.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:15:48.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:15:48.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:15:48.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:15:48.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:15:48.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:15:48.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:15:48.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:15:48.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:15:48.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:15:48.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:15:48.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:15:48.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: ceph osd dump 2026-02-20T17:15:48.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: grep '^pool.*['\'']test['\'']' 2026-02-20T17:15:48.614 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: awk '{ print $2 }' 2026-02-20T17:15:48.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: poolid=1 2026-02-20T17:15:48.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:674: TEST_pg_dump_objects_scrubbed: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:15:48.775 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:15:48.775 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:15:48.775 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.899e-05 s, 17.5 MB/s 2026-02-20T17:15:48.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: seq 1 15 2026-02-20T17:15:48.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj1 testdata.515468 2026-02-20T17:15:48.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj2 testdata.515468 2026-02-20T17:15:48.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj3 testdata.515468 2026-02-20T17:15:48.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj4 testdata.515468 2026-02-20T17:15:48.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj5 testdata.515468 2026-02-20T17:15:48.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj6 testdata.515468 2026-02-20T17:15:48.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj7 testdata.515468 2026-02-20T17:15:48.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj8 testdata.515468 2026-02-20T17:15:48.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj9 testdata.515468 2026-02-20T17:15:48.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:48.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj10 testdata.515468 2026-02-20T17:15:49.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:49.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj11 testdata.515468 2026-02-20T17:15:49.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:49.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj12 testdata.515468 2026-02-20T17:15:49.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:49.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj13 testdata.515468 2026-02-20T17:15:49.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:49.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj14 testdata.515468 2026-02-20T17:15:49.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-20T17:15:49.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj15 testdata.515468 2026-02-20T17:15:49.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: rm -f testdata.515468 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:681: TEST_pg_dump_objects_scrubbed: local pgid=1.0 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:683: TEST_pg_dump_objects_scrubbed: pg_scrub 1.0 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:15:49.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:15:49.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:15:49.121 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:15:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:15:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:15:49.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:15:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:15:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:15:49.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:15:49.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:15:49.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:15:49.411 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:15:49.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:15:49.560 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:15:49.561 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:15:49.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:15:49.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:49.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:15:49.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-20T17:15:49.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-20T17:15:49.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-20T17:15:49.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:49.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:15:49.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-20T17:15:49.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-20T17:15:49.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2026-02-20T17:15:49.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:15:49.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:15:49.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-20T17:15:49.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-20T17:15:49.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-60129542153' 2026-02-20T17:15:49.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:49.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-20T17:15:49.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:49.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:15:49.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-20T17:15:49.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:49.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-20T17:15:49.780 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836496 2026-02-20T17:15:49.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-20T17:15:49.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:15:49.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-20T17:15:49.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:15:50.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:15:50.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:15:51.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836496 2026-02-20T17:15:51.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:51.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-20T17:15:51.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:51.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:15:51.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672973 2026-02-20T17:15:51.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:51.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672973 2026-02-20T17:15:51.121 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672973 2026-02-20T17:15:51.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-20T17:15:51.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:15:51.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672974 -lt 42949672973 2026-02-20T17:15:51.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:15:51.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-20T17:15:51.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:15:51.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:15:51.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-20T17:15:51.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:15:51.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-20T17:15:51.282 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542153 2026-02-20T17:15:51.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-20T17:15:51.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:15:51.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-20T17:15:51.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:15:51.452 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:15:51.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:15:51.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:15:51.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:15:51.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-20T17:15:43.753916+0000 2026-02-20T17:15:51.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-20T17:15:51.830 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-20T17:15:51.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-20T17:15:43.753916+0000 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:15:43.753916+0000 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:15:51.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:15:51.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:15:51.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:15:52.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:15:43.753916+0000 '>' 2026-02-20T17:15:43.753916+0000 2026-02-20T17:15:52.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:15:53.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:15:53.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:15:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:15:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:15:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:15:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:15:53.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:15:53.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:15:52.416821+0000 '>' 2026-02-20T17:15:43.753916+0000 2026-02-20T17:15:53.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:15:53.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: ceph pg 1.0 query 2026-02-20T17:15:53.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: jq .info.stats.objects_scrubbed 2026-02-20T17:15:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: test 15 = 15 2026-02-20T17:15:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:685: TEST_pg_dump_objects_scrubbed: perf_counters td/osd-scrub-test 3 2026-02-20T17:15:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:15:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:15:53.239 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:15:53.241 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:15:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:53.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:15:53.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:53.316 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:53.316 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:53.316 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.316 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.317 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.318 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.319 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.320 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:53.321 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:53.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:53.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:15:53.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.392 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.393 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.394 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.395 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 1, 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.396 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:53.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:53.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:15:53.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.465 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:15:53.466 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:687: TEST_pg_dump_objects_scrubbed: teardown td/osd-scrub-test 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:53.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:53.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:53.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:53.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:53.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:53.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:53.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:53.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:53.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:53.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:53.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:53.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:53.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:53.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:53.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:53.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:53.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:53.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-20T17:15:53.797 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-20T17:15:53.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:15:53.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:53.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:53.801 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:53.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:53.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:53.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:53.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:53.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:53.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:53.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:53.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:53.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:53.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:53.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:53.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:53.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:53.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:53.811 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-20T17:15:53.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-20T17:15:53.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:15:53.811 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_abort -------------------' 2026-02-20T17:15:53.812 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_scrub_abort ------------------- 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:15:53.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:15:53.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:15:53.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:15:53.815 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:15:53.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:15:53.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:15:53.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:15:53.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:15:53.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:15:53.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:15:53.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:15:53.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:15:53.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:15:53.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:15:53.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.823 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:15:53.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:15:53.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:15:53.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:15:53.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:15:53.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.826 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:15:53.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:15:53.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:15:53.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:15:53.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:15:53.828 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_scrub_abort ----------------------- 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_abort -----------------------' 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_abort td/osd-scrub-test 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: TEST_scrub_abort: local dir=td/osd-scrub-test 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: TEST_scrub_abort: _scrub_abort td/osd-scrub-test scrub 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=scrub 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.515468 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test scrub = scrub 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: stopscrub=noscrub 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: check=noscrub 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:15:53.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.851 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:53.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:15:53.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:15:53.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:15:53.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:15:53.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:15:53.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:15:53.882 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.883 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:15:53.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:15:53.884 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:15:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:15:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:15:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:15:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:15:53.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:15:53.940 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:15:53.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:15:53.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:15:53.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:15:53.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:15:53.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:15:53.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:15:53.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:15:53.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:15:54.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:15:54.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:54.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:54.086 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:54.087 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:54.087 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:54.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:54.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:15:54.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:15:54.108 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-20T17:15:54.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:15:54.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:54.114 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:15:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:15:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:15:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:15:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:15:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:15:54.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:15:54.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6e37c613-e802-4dd3-817c-6cecd4fe6f22 2026-02-20T17:15:54.117 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 6e37c613-e802-4dd3-817c-6cecd4fe6f22 2026-02-20T17:15:54.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6e37c613-e802-4dd3-817c-6cecd4fe6f22' 2026-02-20T17:15:54.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:15:54.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKlphpNaBmBxAA9b5lfSrJ7Yx8esHrebyJgw== 2026-02-20T17:15:54.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKlphpNaBmBxAA9b5lfSrJ7Yx8esHrebyJgw=="}' 2026-02-20T17:15:54.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6e37c613-e802-4dd3-817c-6cecd4fe6f22 -i td/osd-scrub-test/0/new.json 2026-02-20T17:15:54.214 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:54.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:15:54.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDKlphpNaBmBxAA9b5lfSrJ7Yx8esHrebyJgw== --osd-uuid 6e37c613-e802-4dd3-817c-6cecd4fe6f22 2026-02-20T17:15:54.241 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:54.238+0000 7fa2a682ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:54.243 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:54.238+0000 7fa2a682ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:54.244 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:54.238+0000 7fa2a682ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:54.244 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:54.242+0000 7fa2a682ca40 -1 bdev(0x55dcdbda6c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:15:54.244 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:54.242+0000 7fa2a682ca40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:15:55.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:15:55.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:15:55.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:15:55.009 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:15:55.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:15:55.111 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:15:55.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:15:55.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:15:55.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:15:55.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:15:55.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:15:55.139 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:55.134+0000 7f1c20a8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:55.141 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:55.138+0000 7f1c20a8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:55.142 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:55.138+0000 7f1c20a8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:55.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:55.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:55.642 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:55.638+0000 7f1c20a8ea40 -1 Falling back to public interface 2026-02-20T17:15:56.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:56.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:56.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:15:56.295 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:56.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:56.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:56.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:56.650 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:56.646+0000 7f1c20a8ea40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:15:57.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:57.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:57.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:15:57.396 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:15:57.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:57.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:15:57.551 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2608389841,v1:127.0.0.1:6801/2608389841] [v2:127.0.0.1:6802/2608389841,v1:127.0.0.1:6803/2608389841] exists,up 6e37c613-e802-4dd3-817c-6cecd4fe6f22 2026-02-20T17:15:57.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:15:57.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:15:57.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:15:57.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:57.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:15:57.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:15:57.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:15:57.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7bf35099-ba79-4248-a822-70e7fa10afd1 2026-02-20T17:15:57.559 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 7bf35099-ba79-4248-a822-70e7fa10afd1 2026-02-20T17:15:57.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7bf35099-ba79-4248-a822-70e7fa10afd1' 2026-02-20T17:15:57.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:15:57.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNlphpxfLYIRAAFQECVttoyo3Af8A1uYrofQ== 2026-02-20T17:15:57.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNlphpxfLYIRAAFQECVttoyo3Af8A1uYrofQ=="}' 2026-02-20T17:15:57.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7bf35099-ba79-4248-a822-70e7fa10afd1 -i td/osd-scrub-test/1/new.json 2026-02-20T17:15:57.720 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:57.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:15:57.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDNlphpxfLYIRAAFQECVttoyo3Af8A1uYrofQ== --osd-uuid 7bf35099-ba79-4248-a822-70e7fa10afd1 2026-02-20T17:15:57.748 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:57.742+0000 7fde7f173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:57.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:57.746+0000 7fde7f173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:57.750 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:57.746+0000 7fde7f173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:57.751 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:57.746+0000 7fde7f173a40 -1 bdev(0x5645002cec00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:15:57.751 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:57.746+0000 7fde7f173a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:15:58.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:15:58.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:15:58.420 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:15:58.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:15:58.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:15:58.617 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:15:58.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:15:58.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:15:58.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:15:58.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:15:58.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:15:58.640 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:58.638+0000 7f474c2d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:58.642 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:58.638+0000 7f474c2d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:58.643 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:58.638+0000 7f474c2d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:15:58.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:58.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:15:58.780 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:15:58.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:58.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:15:58.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:15:59.026 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:59.022+0000 7f474c2d1a40 -1 Falling back to public interface 2026-02-20T17:15:59.613 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:15:59.610+0000 7f474c2d1a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:15:59.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:16:00.095 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2476175118,v1:127.0.0.1:6811/2476175118] [v2:127.0.0.1:6812/2476175118,v1:127.0.0.1:6813/2476175118] exists,up 7bf35099-ba79-4248-a822-70e7fa10afd1 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:16:00.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:16:00.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:16:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:16:00.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36bd41c4-a5dd-4037-8b85-88b63b87514d 2026-02-20T17:16:00.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 36bd41c4-a5dd-4037-8b85-88b63b87514d' 2026-02-20T17:16:00.099 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 36bd41c4-a5dd-4037-8b85-88b63b87514d 2026-02-20T17:16:00.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:16:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQlphpu3aPBhAANo2TkpxbW4AV9VOw7Kc1Xw== 2026-02-20T17:16:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQlphpu3aPBhAANo2TkpxbW4AV9VOw7Kc1Xw=="}' 2026-02-20T17:16:00.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36bd41c4-a5dd-4037-8b85-88b63b87514d -i td/osd-scrub-test/2/new.json 2026-02-20T17:16:00.269 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:16:00.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:16:00.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDQlphpu3aPBhAANo2TkpxbW4AV9VOw7Kc1Xw== --osd-uuid 36bd41c4-a5dd-4037-8b85-88b63b87514d 2026-02-20T17:16:00.297 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:00.294+0000 7f4535267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:00.298 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:00.294+0000 7f4535267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:00.299 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:00.294+0000 7f4535267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:00.299 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:00.294+0000 7f4535267a40 -1 bdev(0x564fb4036c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:16:00.300 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:00.294+0000 7f4535267a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:16:00.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:16:00.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:16:00.959 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:16:00.959 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:16:00.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:16:01.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:16:01.162 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:16:01.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-20T17:16:01.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:16:01.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:16:01.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:16:01.186 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:01.182+0000 7fb599ca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:01.187 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:01.182+0000 7fb599ca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:01.189 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:01.186+0000 7fb599ca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:16:01.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:16:01.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:16:01.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:16:01.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:16:01.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:16:01.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:16:01.326 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:16:01.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:16:01.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:16:01.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:16:01.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:16:01.582 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:01.578+0000 7fb599ca7a40 -1 Falling back to public interface 2026-02-20T17:16:02.140 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:16:02.134+0000 7fb599ca7a40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:16:02.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:16:02.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:16:02.481 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:16:02.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:16:02.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:16:02.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:16:02.632 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/195699475,v1:127.0.0.1:6819/195699475] [v2:127.0.0.1:6820/195699475,v1:127.0.0.1:6821/195699475] exists,up 36bd41c4-a5dd-4037-8b85-88b63b87514d 2026-02-20T17:16:02.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:16:02.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:16:02.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:16:02.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-20T17:16:02.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:16:02.836 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:16:02.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:16:03.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-20T17:16:03.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:16:03.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:16:03.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:16:03.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:16:03.848 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:16:03.849 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:16:03.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:16:03.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:16:03.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:03.960 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:04.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:04.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T17:16:04.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T17:16:04.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T17:16:04.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:04.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:04.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-20T17:16:04.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-20T17:16:04.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-20T17:16:04.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:04.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:04.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-20T17:16:04.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-20T17:16:04.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669 2-55834574851' 2026-02-20T17:16:04.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:04.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T17:16:04.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:04.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:04.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T17:16:04.344 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:04.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T17:16:04.346 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836488 2026-02-20T17:16:04.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T17:16:04.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:04.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T17:16:04.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:04.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-20T17:16:04.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:04.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:04.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-20T17:16:04.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:04.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-20T17:16:04.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-20T17:16:04.521 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705669 2026-02-20T17:16:04.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:04.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-20T17:16:04.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:04.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-20T17:16:04.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:04.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:04.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-20T17:16:04.677 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:04.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-20T17:16:04.679 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574851 2026-02-20T17:16:04.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-20T17:16:04.679 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:04.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-20T17:16:04.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:16:04.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:16:04.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:16:05.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:16:05.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:16:05.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:16:05.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:16:05.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:16:05.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:16:05.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:16:05.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:16:05.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-20T17:16:05.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-20T17:16:05.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-20T17:16:05.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-20T17:16:05.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.515468 bs=1032 count=1 2026-02-20T17:16:05.553 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:16:05.553 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:16:05.553 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.3712e-05 s, 30.6 MB/s 2026-02-20T17:16:05.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-20T17:16:05.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.515468 2026-02-20T17:16:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.515468 2026-02-20T17:16:05.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.515468 2026-02-20T17:16:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.515468 2026-02-20T17:16:05.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.515468 2026-02-20T17:16:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.515468 2026-02-20T17:16:05.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.515468 2026-02-20T17:16:05.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.515468 2026-02-20T17:16:05.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.515468 2026-02-20T17:16:05.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.515468 2026-02-20T17:16:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.515468 2026-02-20T17:16:05.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.515468 2026-02-20T17:16:05.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.515468 2026-02-20T17:16:05.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.515468 2026-02-20T17:16:05.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.515468 2026-02-20T17:16:05.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.515468 2026-02-20T17:16:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.515468 2026-02-20T17:16:05.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.969 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.515468 2026-02-20T17:16:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:05.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.515468 2026-02-20T17:16:06.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.515468 2026-02-20T17:16:06.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.515468 2026-02-20T17:16:06.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.515468 2026-02-20T17:16:06.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.515468 2026-02-20T17:16:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.515468 2026-02-20T17:16:06.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.515468 2026-02-20T17:16:06.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.515468 2026-02-20T17:16:06.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.515468 2026-02-20T17:16:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.515468 2026-02-20T17:16:06.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.515468 2026-02-20T17:16:06.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.515468 2026-02-20T17:16:06.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.515468 2026-02-20T17:16:06.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.515468 2026-02-20T17:16:06.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.515468 2026-02-20T17:16:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.515468 2026-02-20T17:16:06.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.515468 2026-02-20T17:16:06.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.515468 2026-02-20T17:16:06.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.515468 2026-02-20T17:16:06.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.515468 2026-02-20T17:16:06.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.515468 2026-02-20T17:16:06.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.515468 2026-02-20T17:16:06.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.512 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.515468 2026-02-20T17:16:06.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.537 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.515468 2026-02-20T17:16:06.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.515468 2026-02-20T17:16:06.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.515468 2026-02-20T17:16:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.515468 2026-02-20T17:16:06.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.515468 2026-02-20T17:16:06.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.659 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.515468 2026-02-20T17:16:06.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.515468 2026-02-20T17:16:06.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.515468 2026-02-20T17:16:06.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.515468 2026-02-20T17:16:06.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.515468 2026-02-20T17:16:06.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.515468 2026-02-20T17:16:06.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.515468 2026-02-20T17:16:06.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.515468 2026-02-20T17:16:06.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.515468 2026-02-20T17:16:06.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.515468 2026-02-20T17:16:06.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.515468 2026-02-20T17:16:06.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.515468 2026-02-20T17:16:06.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.515468 2026-02-20T17:16:06.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.515468 2026-02-20T17:16:06.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:06.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.515468 2026-02-20T17:16:07.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.018 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.515468 2026-02-20T17:16:07.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.515468 2026-02-20T17:16:07.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.515468 2026-02-20T17:16:07.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.515468 2026-02-20T17:16:07.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.515468 2026-02-20T17:16:07.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.515468 2026-02-20T17:16:07.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.515468 2026-02-20T17:16:07.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.515468 2026-02-20T17:16:07.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.515468 2026-02-20T17:16:07.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.515468 2026-02-20T17:16:07.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.515468 2026-02-20T17:16:07.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.515468 2026-02-20T17:16:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.515468 2026-02-20T17:16:07.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.515468 2026-02-20T17:16:07.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.515468 2026-02-20T17:16:07.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.515468 2026-02-20T17:16:07.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.515468 2026-02-20T17:16:07.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.515468 2026-02-20T17:16:07.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.515468 2026-02-20T17:16:07.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.515468 2026-02-20T17:16:07.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.515468 2026-02-20T17:16:07.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.515468 2026-02-20T17:16:07.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.515468 2026-02-20T17:16:07.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.515468 2026-02-20T17:16:07.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.515468 2026-02-20T17:16:07.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.515468 2026-02-20T17:16:07.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.515468 2026-02-20T17:16:07.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.515468 2026-02-20T17:16:07.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.515468 2026-02-20T17:16:07.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.515468 2026-02-20T17:16:07.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.515468 2026-02-20T17:16:07.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.515468 2026-02-20T17:16:07.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.515468 2026-02-20T17:16:07.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.515468 2026-02-20T17:16:07.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.825 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.515468 2026-02-20T17:16:07.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.515468 2026-02-20T17:16:07.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.515468 2026-02-20T17:16:07.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.515468 2026-02-20T17:16:07.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.515468 2026-02-20T17:16:07.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.515468 2026-02-20T17:16:07.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.515468 2026-02-20T17:16:07.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.515468 2026-02-20T17:16:07.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:07.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.515468 2026-02-20T17:16:08.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.515468 2026-02-20T17:16:08.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.515468 2026-02-20T17:16:08.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.515468 2026-02-20T17:16:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.515468 2026-02-20T17:16:08.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.515468 2026-02-20T17:16:08.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.515468 2026-02-20T17:16:08.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.515468 2026-02-20T17:16:08.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.515468 2026-02-20T17:16:08.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.515468 2026-02-20T17:16:08.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.515468 2026-02-20T17:16:08.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.515468 2026-02-20T17:16:08.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.515468 2026-02-20T17:16:08.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.515468 2026-02-20T17:16:08.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.515468 2026-02-20T17:16:08.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.515468 2026-02-20T17:16:08.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.515468 2026-02-20T17:16:08.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.405 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.515468 2026-02-20T17:16:08.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.515468 2026-02-20T17:16:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.515468 2026-02-20T17:16:08.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.515468 2026-02-20T17:16:08.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.515468 2026-02-20T17:16:08.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.515468 2026-02-20T17:16:08.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.515468 2026-02-20T17:16:08.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.515468 2026-02-20T17:16:08.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.515468 2026-02-20T17:16:08.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.515468 2026-02-20T17:16:08.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.515468 2026-02-20T17:16:08.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.515468 2026-02-20T17:16:08.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.515468 2026-02-20T17:16:08.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.515468 2026-02-20T17:16:08.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.515468 2026-02-20T17:16:08.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.515468 2026-02-20T17:16:08.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.515468 2026-02-20T17:16:08.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.515468 2026-02-20T17:16:08.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.515468 2026-02-20T17:16:08.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.515468 2026-02-20T17:16:08.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.875 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.515468 2026-02-20T17:16:08.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.515468 2026-02-20T17:16:08.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.515468 2026-02-20T17:16:08.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.515468 2026-02-20T17:16:08.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:08.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.515468 2026-02-20T17:16:09.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.515468 2026-02-20T17:16:09.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.515468 2026-02-20T17:16:09.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.515468 2026-02-20T17:16:09.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.515468 2026-02-20T17:16:09.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.515468 2026-02-20T17:16:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.515468 2026-02-20T17:16:09.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.515468 2026-02-20T17:16:09.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.515468 2026-02-20T17:16:09.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.515468 2026-02-20T17:16:09.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.515468 2026-02-20T17:16:09.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.515468 2026-02-20T17:16:09.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.515468 2026-02-20T17:16:09.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.515468 2026-02-20T17:16:09.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.515468 2026-02-20T17:16:09.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.515468 2026-02-20T17:16:09.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.515468 2026-02-20T17:16:09.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.515468 2026-02-20T17:16:09.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.515468 2026-02-20T17:16:09.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.515468 2026-02-20T17:16:09.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.515468 2026-02-20T17:16:09.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.515468 2026-02-20T17:16:09.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.515468 2026-02-20T17:16:09.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.515468 2026-02-20T17:16:09.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.515468 2026-02-20T17:16:09.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.515468 2026-02-20T17:16:09.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.515468 2026-02-20T17:16:09.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.515468 2026-02-20T17:16:09.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.515468 2026-02-20T17:16:09.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.515468 2026-02-20T17:16:09.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.515468 2026-02-20T17:16:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.515468 2026-02-20T17:16:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.515468 2026-02-20T17:16:09.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.515468 2026-02-20T17:16:09.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.515468 2026-02-20T17:16:09.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.515468 2026-02-20T17:16:09.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.515468 2026-02-20T17:16:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.515468 2026-02-20T17:16:09.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.515468 2026-02-20T17:16:09.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.515468 2026-02-20T17:16:09.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.515468 2026-02-20T17:16:09.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.515468 2026-02-20T17:16:09.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.515468 2026-02-20T17:16:09.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.515468 2026-02-20T17:16:09.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:09.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.515468 2026-02-20T17:16:10.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.515468 2026-02-20T17:16:10.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.515468 2026-02-20T17:16:10.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.515468 2026-02-20T17:16:10.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.515468 2026-02-20T17:16:10.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.515468 2026-02-20T17:16:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.515468 2026-02-20T17:16:10.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.515468 2026-02-20T17:16:10.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.515468 2026-02-20T17:16:10.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.515468 2026-02-20T17:16:10.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.515468 2026-02-20T17:16:10.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.515468 2026-02-20T17:16:10.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.515468 2026-02-20T17:16:10.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.515468 2026-02-20T17:16:10.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.515468 2026-02-20T17:16:10.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.515468 2026-02-20T17:16:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.515468 2026-02-20T17:16:10.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.515468 2026-02-20T17:16:10.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.515468 2026-02-20T17:16:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.515468 2026-02-20T17:16:10.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.515468 2026-02-20T17:16:10.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.515468 2026-02-20T17:16:10.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.515468 2026-02-20T17:16:10.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.515468 2026-02-20T17:16:10.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.515468 2026-02-20T17:16:10.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.515468 2026-02-20T17:16:10.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.515468 2026-02-20T17:16:10.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.515468 2026-02-20T17:16:10.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.515468 2026-02-20T17:16:10.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.515468 2026-02-20T17:16:10.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.515468 2026-02-20T17:16:10.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.515468 2026-02-20T17:16:10.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.515468 2026-02-20T17:16:10.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.515468 2026-02-20T17:16:10.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.515468 2026-02-20T17:16:10.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.843 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.515468 2026-02-20T17:16:10.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.515468 2026-02-20T17:16:10.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.515468 2026-02-20T17:16:10.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.515468 2026-02-20T17:16:10.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.515468 2026-02-20T17:16:10.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.515468 2026-02-20T17:16:10.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.515468 2026-02-20T17:16:10.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:10.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.515468 2026-02-20T17:16:11.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.515468 2026-02-20T17:16:11.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.515468 2026-02-20T17:16:11.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.515468 2026-02-20T17:16:11.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.515468 2026-02-20T17:16:11.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.515468 2026-02-20T17:16:11.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.515468 2026-02-20T17:16:11.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.515468 2026-02-20T17:16:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.515468 2026-02-20T17:16:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.515468 2026-02-20T17:16:11.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.515468 2026-02-20T17:16:11.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.515468 2026-02-20T17:16:11.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.515468 2026-02-20T17:16:11.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.515468 2026-02-20T17:16:11.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.515468 2026-02-20T17:16:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.515468 2026-02-20T17:16:11.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.515468 2026-02-20T17:16:11.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.515468 2026-02-20T17:16:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.515468 2026-02-20T17:16:11.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.434 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.515468 2026-02-20T17:16:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.515468 2026-02-20T17:16:11.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.515468 2026-02-20T17:16:11.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.515468 2026-02-20T17:16:11.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.515468 2026-02-20T17:16:11.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.515468 2026-02-20T17:16:11.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.515468 2026-02-20T17:16:11.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.515468 2026-02-20T17:16:11.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.515468 2026-02-20T17:16:11.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.515468 2026-02-20T17:16:11.671 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.515468 2026-02-20T17:16:11.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.515468 2026-02-20T17:16:11.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.515468 2026-02-20T17:16:11.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.515468 2026-02-20T17:16:11.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.515468 2026-02-20T17:16:11.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.515468 2026-02-20T17:16:11.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.515468 2026-02-20T17:16:11.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.515468 2026-02-20T17:16:11.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.515468 2026-02-20T17:16:11.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.515468 2026-02-20T17:16:11.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.515468 2026-02-20T17:16:11.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.924 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.515468 2026-02-20T17:16:11.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.515468 2026-02-20T17:16:11.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.515468 2026-02-20T17:16:11.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:11.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.515468 2026-02-20T17:16:12.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.515468 2026-02-20T17:16:12.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.515468 2026-02-20T17:16:12.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.515468 2026-02-20T17:16:12.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.515468 2026-02-20T17:16:12.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.515468 2026-02-20T17:16:12.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.515468 2026-02-20T17:16:12.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.515468 2026-02-20T17:16:12.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.515468 2026-02-20T17:16:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.515468 2026-02-20T17:16:12.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.515468 2026-02-20T17:16:12.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.515468 2026-02-20T17:16:12.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.515468 2026-02-20T17:16:12.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.515468 2026-02-20T17:16:12.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.515468 2026-02-20T17:16:12.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.515468 2026-02-20T17:16:12.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.515468 2026-02-20T17:16:12.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.515468 2026-02-20T17:16:12.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.515468 2026-02-20T17:16:12.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.441 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.515468 2026-02-20T17:16:12.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.515468 2026-02-20T17:16:12.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.515468 2026-02-20T17:16:12.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.515468 2026-02-20T17:16:12.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.515468 2026-02-20T17:16:12.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.515468 2026-02-20T17:16:12.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.515468 2026-02-20T17:16:12.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.515468 2026-02-20T17:16:12.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.515468 2026-02-20T17:16:12.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.515468 2026-02-20T17:16:12.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.515468 2026-02-20T17:16:12.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.515468 2026-02-20T17:16:12.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.515468 2026-02-20T17:16:12.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.515468 2026-02-20T17:16:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.515468 2026-02-20T17:16:12.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.515468 2026-02-20T17:16:12.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.515468 2026-02-20T17:16:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.515468 2026-02-20T17:16:12.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.855 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.515468 2026-02-20T17:16:12.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.515468 2026-02-20T17:16:12.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.515468 2026-02-20T17:16:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.515468 2026-02-20T17:16:12.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.515468 2026-02-20T17:16:12.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.515468 2026-02-20T17:16:12.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:12.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.515468 2026-02-20T17:16:13.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.515468 2026-02-20T17:16:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.515468 2026-02-20T17:16:13.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.515468 2026-02-20T17:16:13.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.515468 2026-02-20T17:16:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.515468 2026-02-20T17:16:13.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.515468 2026-02-20T17:16:13.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.515468 2026-02-20T17:16:13.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.515468 2026-02-20T17:16:13.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.515468 2026-02-20T17:16:13.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.515468 2026-02-20T17:16:13.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.515468 2026-02-20T17:16:13.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.515468 2026-02-20T17:16:13.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.515468 2026-02-20T17:16:13.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.515468 2026-02-20T17:16:13.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.515468 2026-02-20T17:16:13.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.515468 2026-02-20T17:16:13.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.515468 2026-02-20T17:16:13.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.515468 2026-02-20T17:16:13.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.515468 2026-02-20T17:16:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.515468 2026-02-20T17:16:13.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.515468 2026-02-20T17:16:13.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.515468 2026-02-20T17:16:13.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.515468 2026-02-20T17:16:13.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.515468 2026-02-20T17:16:13.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.515468 2026-02-20T17:16:13.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.515468 2026-02-20T17:16:13.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.515468 2026-02-20T17:16:13.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.515468 2026-02-20T17:16:13.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.515468 2026-02-20T17:16:13.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.515468 2026-02-20T17:16:13.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.515468 2026-02-20T17:16:13.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.515468 2026-02-20T17:16:13.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.515468 2026-02-20T17:16:13.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.515468 2026-02-20T17:16:13.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.515468 2026-02-20T17:16:13.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.515468 2026-02-20T17:16:13.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.515468 2026-02-20T17:16:13.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.515468 2026-02-20T17:16:13.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.515468 2026-02-20T17:16:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.515468 2026-02-20T17:16:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.515468 2026-02-20T17:16:13.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.515468 2026-02-20T17:16:13.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:13.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.515468 2026-02-20T17:16:14.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.515468 2026-02-20T17:16:14.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.515468 2026-02-20T17:16:14.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.515468 2026-02-20T17:16:14.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.515468 2026-02-20T17:16:14.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.515468 2026-02-20T17:16:14.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.515468 2026-02-20T17:16:14.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.169 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.515468 2026-02-20T17:16:14.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.515468 2026-02-20T17:16:14.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.515468 2026-02-20T17:16:14.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.515468 2026-02-20T17:16:14.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.515468 2026-02-20T17:16:14.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.515468 2026-02-20T17:16:14.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.515468 2026-02-20T17:16:14.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.515468 2026-02-20T17:16:14.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.366 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.515468 2026-02-20T17:16:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.515468 2026-02-20T17:16:14.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.515468 2026-02-20T17:16:14.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.515468 2026-02-20T17:16:14.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.466 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.515468 2026-02-20T17:16:14.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.515468 2026-02-20T17:16:14.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.515468 2026-02-20T17:16:14.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.515468 2026-02-20T17:16:14.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.515468 2026-02-20T17:16:14.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.515468 2026-02-20T17:16:14.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.515468 2026-02-20T17:16:14.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.515468 2026-02-20T17:16:14.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.515468 2026-02-20T17:16:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.515468 2026-02-20T17:16:14.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.515468 2026-02-20T17:16:14.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.515468 2026-02-20T17:16:14.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.515468 2026-02-20T17:16:14.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.515468 2026-02-20T17:16:14.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.515468 2026-02-20T17:16:14.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.515468 2026-02-20T17:16:14.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.515468 2026-02-20T17:16:14.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.515468 2026-02-20T17:16:14.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.515468 2026-02-20T17:16:14.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.515468 2026-02-20T17:16:14.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.515468 2026-02-20T17:16:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.515468 2026-02-20T17:16:14.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.515468 2026-02-20T17:16:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:14.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.515468 2026-02-20T17:16:15.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.515468 2026-02-20T17:16:15.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.515468 2026-02-20T17:16:15.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.515468 2026-02-20T17:16:15.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.089 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.515468 2026-02-20T17:16:15.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.515468 2026-02-20T17:16:15.138 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.515468 2026-02-20T17:16:15.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.515468 2026-02-20T17:16:15.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.515468 2026-02-20T17:16:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.515468 2026-02-20T17:16:15.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.515468 2026-02-20T17:16:15.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.515468 2026-02-20T17:16:15.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.515468 2026-02-20T17:16:15.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.515468 2026-02-20T17:16:15.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.515468 2026-02-20T17:16:15.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.515468 2026-02-20T17:16:15.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.361 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.515468 2026-02-20T17:16:15.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.515468 2026-02-20T17:16:15.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.515468 2026-02-20T17:16:15.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.515468 2026-02-20T17:16:15.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.515468 2026-02-20T17:16:15.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.515468 2026-02-20T17:16:15.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.515468 2026-02-20T17:16:15.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.515468 2026-02-20T17:16:15.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.515468 2026-02-20T17:16:15.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.515468 2026-02-20T17:16:15.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.515468 2026-02-20T17:16:15.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.515468 2026-02-20T17:16:15.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.515468 2026-02-20T17:16:15.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.515468 2026-02-20T17:16:15.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.515468 2026-02-20T17:16:15.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.515468 2026-02-20T17:16:15.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.515468 2026-02-20T17:16:15.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.515468 2026-02-20T17:16:15.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.515468 2026-02-20T17:16:15.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.515468 2026-02-20T17:16:15.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.832 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.515468 2026-02-20T17:16:15.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.515468 2026-02-20T17:16:15.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.515468 2026-02-20T17:16:15.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.515468 2026-02-20T17:16:15.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.515468 2026-02-20T17:16:15.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.515468 2026-02-20T17:16:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.515468 2026-02-20T17:16:16.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.515468 2026-02-20T17:16:16.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.029 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.515468 2026-02-20T17:16:16.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.054 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.515468 2026-02-20T17:16:16.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.515468 2026-02-20T17:16:16.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.515468 2026-02-20T17:16:16.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.515468 2026-02-20T17:16:16.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.515468 2026-02-20T17:16:16.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.515468 2026-02-20T17:16:16.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.515468 2026-02-20T17:16:16.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.515468 2026-02-20T17:16:16.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.515468 2026-02-20T17:16:16.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.515468 2026-02-20T17:16:16.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.515468 2026-02-20T17:16:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.515468 2026-02-20T17:16:16.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.515468 2026-02-20T17:16:16.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.515468 2026-02-20T17:16:16.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.515468 2026-02-20T17:16:16.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.404 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.515468 2026-02-20T17:16:16.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.515468 2026-02-20T17:16:16.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.515468 2026-02-20T17:16:16.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.515468 2026-02-20T17:16:16.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.515468 2026-02-20T17:16:16.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.515468 2026-02-20T17:16:16.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.515468 2026-02-20T17:16:16.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.515468 2026-02-20T17:16:16.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.515468 2026-02-20T17:16:16.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.515468 2026-02-20T17:16:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.515468 2026-02-20T17:16:16.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.515468 2026-02-20T17:16:16.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.515468 2026-02-20T17:16:16.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.515468 2026-02-20T17:16:16.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.515468 2026-02-20T17:16:16.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.515468 2026-02-20T17:16:16.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.515468 2026-02-20T17:16:16.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.515468 2026-02-20T17:16:16.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.515468 2026-02-20T17:16:16.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.515468 2026-02-20T17:16:16.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.515468 2026-02-20T17:16:16.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.874 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.515468 2026-02-20T17:16:16.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.515468 2026-02-20T17:16:16.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.515468 2026-02-20T17:16:16.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.515468 2026-02-20T17:16:16.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.515468 2026-02-20T17:16:16.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:16.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.515468 2026-02-20T17:16:17.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.515468 2026-02-20T17:16:17.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.515468 2026-02-20T17:16:17.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.515468 2026-02-20T17:16:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.515468 2026-02-20T17:16:17.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.515468 2026-02-20T17:16:17.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.125 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.515468 2026-02-20T17:16:17.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.515468 2026-02-20T17:16:17.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.515468 2026-02-20T17:16:17.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.199 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.515468 2026-02-20T17:16:17.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.515468 2026-02-20T17:16:17.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.515468 2026-02-20T17:16:17.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.515468 2026-02-20T17:16:17.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.515468 2026-02-20T17:16:17.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.515468 2026-02-20T17:16:17.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.515468 2026-02-20T17:16:17.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.515468 2026-02-20T17:16:17.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.515468 2026-02-20T17:16:17.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.515468 2026-02-20T17:16:17.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.515468 2026-02-20T17:16:17.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.515468 2026-02-20T17:16:17.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.515468 2026-02-20T17:16:17.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.515468 2026-02-20T17:16:17.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.515468 2026-02-20T17:16:17.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.515468 2026-02-20T17:16:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.515468 2026-02-20T17:16:17.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.515468 2026-02-20T17:16:17.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.515468 2026-02-20T17:16:17.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.515468 2026-02-20T17:16:17.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.515468 2026-02-20T17:16:17.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.515468 2026-02-20T17:16:17.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.515468 2026-02-20T17:16:17.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.515468 2026-02-20T17:16:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.515468 2026-02-20T17:16:17.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.515468 2026-02-20T17:16:17.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.515468 2026-02-20T17:16:17.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.840 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.515468 2026-02-20T17:16:17.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.515468 2026-02-20T17:16:17.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.515468 2026-02-20T17:16:17.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.515468 2026-02-20T17:16:17.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.515468 2026-02-20T17:16:17.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.515468 2026-02-20T17:16:17.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:17.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.515468 2026-02-20T17:16:18.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.515468 2026-02-20T17:16:18.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.515468 2026-02-20T17:16:18.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.053 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.515468 2026-02-20T17:16:18.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.515468 2026-02-20T17:16:18.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.515468 2026-02-20T17:16:18.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.515468 2026-02-20T17:16:18.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.515468 2026-02-20T17:16:18.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.515468 2026-02-20T17:16:18.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.515468 2026-02-20T17:16:18.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.515468 2026-02-20T17:16:18.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.515468 2026-02-20T17:16:18.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.515468 2026-02-20T17:16:18.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.515468 2026-02-20T17:16:18.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.515468 2026-02-20T17:16:18.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.515468 2026-02-20T17:16:18.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.515468 2026-02-20T17:16:18.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.515468 2026-02-20T17:16:18.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.515468 2026-02-20T17:16:18.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.515468 2026-02-20T17:16:18.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.515468 2026-02-20T17:16:18.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.515468 2026-02-20T17:16:18.502 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.515468 2026-02-20T17:16:18.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.515468 2026-02-20T17:16:18.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.515468 2026-02-20T17:16:18.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.515468 2026-02-20T17:16:18.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.515468 2026-02-20T17:16:18.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.515468 2026-02-20T17:16:18.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.515468 2026-02-20T17:16:18.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.515468 2026-02-20T17:16:18.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.680 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.515468 2026-02-20T17:16:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.515468 2026-02-20T17:16:18.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.515468 2026-02-20T17:16:18.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.515468 2026-02-20T17:16:18.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.515468 2026-02-20T17:16:18.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.515468 2026-02-20T17:16:18.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.515468 2026-02-20T17:16:18.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.515468 2026-02-20T17:16:18.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.515468 2026-02-20T17:16:18.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.515468 2026-02-20T17:16:18.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.515468 2026-02-20T17:16:18.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.944 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.515468 2026-02-20T17:16:18.967 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.515468 2026-02-20T17:16:18.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:18.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.515468 2026-02-20T17:16:19.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.515468 2026-02-20T17:16:19.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.515468 2026-02-20T17:16:19.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.515468 2026-02-20T17:16:19.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.515468 2026-02-20T17:16:19.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.515468 2026-02-20T17:16:19.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.515468 2026-02-20T17:16:19.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.515468 2026-02-20T17:16:19.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.515468 2026-02-20T17:16:19.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.515468 2026-02-20T17:16:19.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.515468 2026-02-20T17:16:19.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.515468 2026-02-20T17:16:19.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.515468 2026-02-20T17:16:19.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.515468 2026-02-20T17:16:19.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.515468 2026-02-20T17:16:19.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.515468 2026-02-20T17:16:19.344 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.515468 2026-02-20T17:16:19.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.515468 2026-02-20T17:16:19.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.515468 2026-02-20T17:16:19.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.515468 2026-02-20T17:16:19.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.515468 2026-02-20T17:16:19.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.515468 2026-02-20T17:16:19.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.515468 2026-02-20T17:16:19.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.515468 2026-02-20T17:16:19.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.515468 2026-02-20T17:16:19.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.515468 2026-02-20T17:16:19.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.515468 2026-02-20T17:16:19.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.515468 2026-02-20T17:16:19.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.515468 2026-02-20T17:16:19.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.515468 2026-02-20T17:16:19.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.515468 2026-02-20T17:16:19.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.515468 2026-02-20T17:16:19.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.515468 2026-02-20T17:16:19.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.515468 2026-02-20T17:16:19.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.515468 2026-02-20T17:16:19.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.515468 2026-02-20T17:16:19.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.515468 2026-02-20T17:16:19.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.515468 2026-02-20T17:16:19.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.515468 2026-02-20T17:16:19.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.515468 2026-02-20T17:16:19.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.515468 2026-02-20T17:16:19.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.515468 2026-02-20T17:16:19.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.515468 2026-02-20T17:16:19.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:19.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.515468 2026-02-20T17:16:20.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.001 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.515468 2026-02-20T17:16:20.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.515468 2026-02-20T17:16:20.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.515468 2026-02-20T17:16:20.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.515468 2026-02-20T17:16:20.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.515468 2026-02-20T17:16:20.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.515468 2026-02-20T17:16:20.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.515468 2026-02-20T17:16:20.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.515468 2026-02-20T17:16:20.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.515468 2026-02-20T17:16:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.515468 2026-02-20T17:16:20.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.515468 2026-02-20T17:16:20.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.515468 2026-02-20T17:16:20.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.515468 2026-02-20T17:16:20.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.515468 2026-02-20T17:16:20.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.515468 2026-02-20T17:16:20.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.515468 2026-02-20T17:16:20.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.515468 2026-02-20T17:16:20.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.515468 2026-02-20T17:16:20.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.515468 2026-02-20T17:16:20.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.515468 2026-02-20T17:16:20.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.515468 2026-02-20T17:16:20.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.515468 2026-02-20T17:16:20.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.515468 2026-02-20T17:16:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.515468 2026-02-20T17:16:20.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.515468 2026-02-20T17:16:20.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.571 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.515468 2026-02-20T17:16:20.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.515468 2026-02-20T17:16:20.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.515468 2026-02-20T17:16:20.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.515468 2026-02-20T17:16:20.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.515468 2026-02-20T17:16:20.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.515468 2026-02-20T17:16:20.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.515468 2026-02-20T17:16:20.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.515468 2026-02-20T17:16:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.762 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.515468 2026-02-20T17:16:20.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.787 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.515468 2026-02-20T17:16:20.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.515468 2026-02-20T17:16:20.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.515468 2026-02-20T17:16:20.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.856 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.515468 2026-02-20T17:16:20.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.515468 2026-02-20T17:16:20.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.515468 2026-02-20T17:16:20.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.515468 2026-02-20T17:16:20.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.515468 2026-02-20T17:16:20.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.515468 2026-02-20T17:16:20.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:20.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.515468 2026-02-20T17:16:21.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.515468 2026-02-20T17:16:21.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.515468 2026-02-20T17:16:21.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.515468 2026-02-20T17:16:21.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.515468 2026-02-20T17:16:21.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.515468 2026-02-20T17:16:21.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.515468 2026-02-20T17:16:21.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.515468 2026-02-20T17:16:21.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.515468 2026-02-20T17:16:21.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.515468 2026-02-20T17:16:21.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.515468 2026-02-20T17:16:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.515468 2026-02-20T17:16:21.253 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.515468 2026-02-20T17:16:21.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.515468 2026-02-20T17:16:21.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.515468 2026-02-20T17:16:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.515468 2026-02-20T17:16:21.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.515468 2026-02-20T17:16:21.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.515468 2026-02-20T17:16:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.515468 2026-02-20T17:16:21.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.515468 2026-02-20T17:16:21.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.515468 2026-02-20T17:16:21.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.515468 2026-02-20T17:16:21.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.515468 2026-02-20T17:16:21.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.515468 2026-02-20T17:16:21.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.515468 2026-02-20T17:16:21.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.515468 2026-02-20T17:16:21.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.515468 2026-02-20T17:16:21.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.515468 2026-02-20T17:16:21.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.515468 2026-02-20T17:16:21.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.515468 2026-02-20T17:16:21.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.515468 2026-02-20T17:16:21.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.515468 2026-02-20T17:16:21.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.515468 2026-02-20T17:16:21.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.515468 2026-02-20T17:16:21.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.515468 2026-02-20T17:16:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.515468 2026-02-20T17:16:21.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.515468 2026-02-20T17:16:21.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.515468 2026-02-20T17:16:21.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.515468 2026-02-20T17:16:21.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.515468 2026-02-20T17:16:21.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.515468 2026-02-20T17:16:21.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.515468 2026-02-20T17:16:21.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.515468 2026-02-20T17:16:21.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:21.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.515468 2026-02-20T17:16:22.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.000 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.515468 2026-02-20T17:16:22.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.515468 2026-02-20T17:16:22.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.515468 2026-02-20T17:16:22.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.515468 2026-02-20T17:16:22.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.515468 2026-02-20T17:16:22.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.515468 2026-02-20T17:16:22.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.515468 2026-02-20T17:16:22.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.515468 2026-02-20T17:16:22.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.515468 2026-02-20T17:16:22.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.515468 2026-02-20T17:16:22.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.515468 2026-02-20T17:16:22.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.515468 2026-02-20T17:16:22.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.515468 2026-02-20T17:16:22.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.515468 2026-02-20T17:16:22.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.515468 2026-02-20T17:16:22.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.515468 2026-02-20T17:16:22.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.515468 2026-02-20T17:16:22.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.407 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.515468 2026-02-20T17:16:22.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.515468 2026-02-20T17:16:22.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.515468 2026-02-20T17:16:22.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.515468 2026-02-20T17:16:22.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.515468 2026-02-20T17:16:22.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.515468 2026-02-20T17:16:22.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.515468 2026-02-20T17:16:22.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.515468 2026-02-20T17:16:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.515468 2026-02-20T17:16:22.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.515468 2026-02-20T17:16:22.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.650 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.515468 2026-02-20T17:16:22.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.515468 2026-02-20T17:16:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.515468 2026-02-20T17:16:22.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.515468 2026-02-20T17:16:22.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.515468 2026-02-20T17:16:22.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.515468 2026-02-20T17:16:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.515468 2026-02-20T17:16:22.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.515468 2026-02-20T17:16:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.515468 2026-02-20T17:16:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.515468 2026-02-20T17:16:22.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.515468 2026-02-20T17:16:22.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.515468 2026-02-20T17:16:22.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.515468 2026-02-20T17:16:22.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.515468 2026-02-20T17:16:22.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.977 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.515468 2026-02-20T17:16:22.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:22.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.515468 2026-02-20T17:16:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.515468 2026-02-20T17:16:23.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.515468 2026-02-20T17:16:23.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.515468 2026-02-20T17:16:23.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.515468 2026-02-20T17:16:23.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.515468 2026-02-20T17:16:23.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.515468 2026-02-20T17:16:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.515468 2026-02-20T17:16:23.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.515468 2026-02-20T17:16:23.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.205 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.515468 2026-02-20T17:16:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.515468 2026-02-20T17:16:23.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.515468 2026-02-20T17:16:23.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.515468 2026-02-20T17:16:23.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.515468 2026-02-20T17:16:23.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.515468 2026-02-20T17:16:23.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.515468 2026-02-20T17:16:23.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.515468 2026-02-20T17:16:23.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.515468 2026-02-20T17:16:23.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.409 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.515468 2026-02-20T17:16:23.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.515468 2026-02-20T17:16:23.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.515468 2026-02-20T17:16:23.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.515468 2026-02-20T17:16:23.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.515468 2026-02-20T17:16:23.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.515468 2026-02-20T17:16:23.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.515468 2026-02-20T17:16:23.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.515468 2026-02-20T17:16:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.515468 2026-02-20T17:16:23.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.515468 2026-02-20T17:16:23.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.515468 2026-02-20T17:16:23.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.515468 2026-02-20T17:16:23.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.515468 2026-02-20T17:16:23.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.515468 2026-02-20T17:16:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.515468 2026-02-20T17:16:23.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.515468 2026-02-20T17:16:23.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.515468 2026-02-20T17:16:23.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.812 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.515468 2026-02-20T17:16:23.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.836 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.515468 2026-02-20T17:16:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.861 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.515468 2026-02-20T17:16:23.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.515468 2026-02-20T17:16:23.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.515468 2026-02-20T17:16:23.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.515468 2026-02-20T17:16:23.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.515468 2026-02-20T17:16:23.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:23.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.515468 2026-02-20T17:16:24.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.515468 2026-02-20T17:16:24.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.515468 2026-02-20T17:16:24.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.515468 2026-02-20T17:16:24.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.515468 2026-02-20T17:16:24.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.515468 2026-02-20T17:16:24.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.515468 2026-02-20T17:16:24.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.515468 2026-02-20T17:16:24.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.515468 2026-02-20T17:16:24.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.515468 2026-02-20T17:16:24.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.515468 2026-02-20T17:16:24.240 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.515468 2026-02-20T17:16:24.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.515468 2026-02-20T17:16:24.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.515468 2026-02-20T17:16:24.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.515468 2026-02-20T17:16:24.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.515468 2026-02-20T17:16:24.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.515468 2026-02-20T17:16:24.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.515468 2026-02-20T17:16:24.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.515468 2026-02-20T17:16:24.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.515468 2026-02-20T17:16:24.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.515468 2026-02-20T17:16:24.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.515468 2026-02-20T17:16:24.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.515468 2026-02-20T17:16:24.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.515468 2026-02-20T17:16:24.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.515468 2026-02-20T17:16:24.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.515468 2026-02-20T17:16:24.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.515468 2026-02-20T17:16:24.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.619 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.515468 2026-02-20T17:16:24.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.515468 2026-02-20T17:16:24.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.515468 2026-02-20T17:16:24.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.515468 2026-02-20T17:16:24.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.515468 2026-02-20T17:16:24.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.743 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.515468 2026-02-20T17:16:24.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.515468 2026-02-20T17:16:24.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.515468 2026-02-20T17:16:24.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.515468 2026-02-20T17:16:24.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.515468 2026-02-20T17:16:24.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.515468 2026-02-20T17:16:24.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.515468 2026-02-20T17:16:24.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.515468 2026-02-20T17:16:24.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.515468 2026-02-20T17:16:24.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.515468 2026-02-20T17:16:24.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:24.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.515468 2026-02-20T17:16:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.515468 2026-02-20T17:16:25.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.515468 2026-02-20T17:16:25.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.052 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.515468 2026-02-20T17:16:25.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.515468 2026-02-20T17:16:25.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.515468 2026-02-20T17:16:25.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.515468 2026-02-20T17:16:25.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.515468 2026-02-20T17:16:25.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.515468 2026-02-20T17:16:25.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.176 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.515468 2026-02-20T17:16:25.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.515468 2026-02-20T17:16:25.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.515468 2026-02-20T17:16:25.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.515468 2026-02-20T17:16:25.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.515468 2026-02-20T17:16:25.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.515468 2026-02-20T17:16:25.313 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.515468 2026-02-20T17:16:25.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.515468 2026-02-20T17:16:25.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.515468 2026-02-20T17:16:25.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.515468 2026-02-20T17:16:25.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.515468 2026-02-20T17:16:25.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.515468 2026-02-20T17:16:25.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.515468 2026-02-20T17:16:25.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.515468 2026-02-20T17:16:25.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.515468 2026-02-20T17:16:25.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.515468 2026-02-20T17:16:25.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.515468 2026-02-20T17:16:25.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.515468 2026-02-20T17:16:25.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.515468 2026-02-20T17:16:25.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.515468 2026-02-20T17:16:25.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.515468 2026-02-20T17:16:25.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.515468 2026-02-20T17:16:25.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.515468 2026-02-20T17:16:25.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.693 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.515468 2026-02-20T17:16:25.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.515468 2026-02-20T17:16:25.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.515468 2026-02-20T17:16:25.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.515468 2026-02-20T17:16:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.515468 2026-02-20T17:16:25.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.515468 2026-02-20T17:16:25.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.515468 2026-02-20T17:16:25.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.515468 2026-02-20T17:16:25.883 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.515468 2026-02-20T17:16:25.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.515468 2026-02-20T17:16:25.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.515468 2026-02-20T17:16:25.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.515468 2026-02-20T17:16:25.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.515468 2026-02-20T17:16:25.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:25.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.515468 2026-02-20T17:16:26.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.515468 2026-02-20T17:16:26.047 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.515468 2026-02-20T17:16:26.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.515468 2026-02-20T17:16:26.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.515468 2026-02-20T17:16:26.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.515468 2026-02-20T17:16:26.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.515468 2026-02-20T17:16:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.515468 2026-02-20T17:16:26.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.171 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.515468 2026-02-20T17:16:26.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.515468 2026-02-20T17:16:26.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.515468 2026-02-20T17:16:26.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.515468 2026-02-20T17:16:26.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.515468 2026-02-20T17:16:26.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.515468 2026-02-20T17:16:26.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.318 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.515468 2026-02-20T17:16:26.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.343 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.515468 2026-02-20T17:16:26.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.515468 2026-02-20T17:16:26.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.515468 2026-02-20T17:16:26.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.515468 2026-02-20T17:16:26.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.515468 2026-02-20T17:16:26.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.515468 2026-02-20T17:16:26.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.515468 2026-02-20T17:16:26.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.517 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.515468 2026-02-20T17:16:26.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.515468 2026-02-20T17:16:26.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.515468 2026-02-20T17:16:26.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.515468 2026-02-20T17:16:26.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.515468 2026-02-20T17:16:26.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.515468 2026-02-20T17:16:26.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.515468 2026-02-20T17:16:26.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.515468 2026-02-20T17:16:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.515468 2026-02-20T17:16:26.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.515468 2026-02-20T17:16:26.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.515468 2026-02-20T17:16:26.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.515468 2026-02-20T17:16:26.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.515468 2026-02-20T17:16:26.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.515468 2026-02-20T17:16:26.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.515468 2026-02-20T17:16:26.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.515468 2026-02-20T17:16:26.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.515468 2026-02-20T17:16:26.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.515468 2026-02-20T17:16:26.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.515468 2026-02-20T17:16:26.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.515468 2026-02-20T17:16:26.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:26.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.515468 2026-02-20T17:16:27.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.515468 2026-02-20T17:16:27.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.515468 2026-02-20T17:16:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.515468 2026-02-20T17:16:27.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.515468 2026-02-20T17:16:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.515468 2026-02-20T17:16:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.515468 2026-02-20T17:16:27.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.515468 2026-02-20T17:16:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.515468 2026-02-20T17:16:27.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.515468 2026-02-20T17:16:27.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.515468 2026-02-20T17:16:27.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.515468 2026-02-20T17:16:27.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.515468 2026-02-20T17:16:27.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.515468 2026-02-20T17:16:27.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.515468 2026-02-20T17:16:27.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.515468 2026-02-20T17:16:27.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.515468 2026-02-20T17:16:27.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.515468 2026-02-20T17:16:27.422 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.423 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.515468 2026-02-20T17:16:27.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.515468 2026-02-20T17:16:27.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.515468 2026-02-20T17:16:27.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.515468 2026-02-20T17:16:27.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.510 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.515468 2026-02-20T17:16:27.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.515468 2026-02-20T17:16:27.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.515468 2026-02-20T17:16:27.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.515468 2026-02-20T17:16:27.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.515468 2026-02-20T17:16:27.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.515468 2026-02-20T17:16:27.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.515468 2026-02-20T17:16:27.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.515468 2026-02-20T17:16:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.515468 2026-02-20T17:16:27.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.515468 2026-02-20T17:16:27.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.515468 2026-02-20T17:16:27.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.515468 2026-02-20T17:16:27.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.515468 2026-02-20T17:16:27.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.515468 2026-02-20T17:16:27.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.515468 2026-02-20T17:16:27.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.515468 2026-02-20T17:16:27.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.515468 2026-02-20T17:16:27.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.515468 2026-02-20T17:16:27.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.515468 2026-02-20T17:16:27.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.515468 2026-02-20T17:16:27.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:27.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.515468 2026-02-20T17:16:28.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.515468 2026-02-20T17:16:28.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.515468 2026-02-20T17:16:28.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.515468 2026-02-20T17:16:28.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.515468 2026-02-20T17:16:28.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.515468 2026-02-20T17:16:28.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.515468 2026-02-20T17:16:28.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.515468 2026-02-20T17:16:28.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.515468 2026-02-20T17:16:28.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.515468 2026-02-20T17:16:28.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.204 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.515468 2026-02-20T17:16:28.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.224 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.515468 2026-02-20T17:16:28.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.515468 2026-02-20T17:16:28.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.515468 2026-02-20T17:16:28.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.515468 2026-02-20T17:16:28.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.515468 2026-02-20T17:16:28.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.515468 2026-02-20T17:16:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.515468 2026-02-20T17:16:28.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.515468 2026-02-20T17:16:28.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.401 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.515468 2026-02-20T17:16:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.425 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.515468 2026-02-20T17:16:28.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.515468 2026-02-20T17:16:28.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.476 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.515468 2026-02-20T17:16:28.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.515468 2026-02-20T17:16:28.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.515468 2026-02-20T17:16:28.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.515468 2026-02-20T17:16:28.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.515468 2026-02-20T17:16:28.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.515468 2026-02-20T17:16:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.515468 2026-02-20T17:16:28.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.515468 2026-02-20T17:16:28.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.515468 2026-02-20T17:16:28.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.515468 2026-02-20T17:16:28.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.515468 2026-02-20T17:16:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.515468 2026-02-20T17:16:28.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.515468 2026-02-20T17:16:28.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.515468 2026-02-20T17:16:28.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.515468 2026-02-20T17:16:28.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.515468 2026-02-20T17:16:28.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.515468 2026-02-20T17:16:28.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.515468 2026-02-20T17:16:28.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.515468 2026-02-20T17:16:28.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-20T17:16:28.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.515468 2026-02-20T17:16:28.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.515468 2026-02-20T17:16:28.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-20T17:16:28.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:16:28.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:16:28.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:16:28.954 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:16:29.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-20T17:16:29.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-20T17:16:29.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-scrub 2026-02-20T17:16:29.176 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:16:29.176 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:16:29.177 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:16:29.177 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-18T17:16:29.176092+0000" 2026-02-20T17:16:29.177 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:16:29.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-20T17:16:29.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-20T17:16:29.184 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-20T17:16:29.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:29.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-20T17:16:29.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:29.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:29.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:29.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-20T17:16:29.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-20T17:16:29.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-20T17:16:29.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:29.415 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:29.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705695 2026-02-20T17:16:29.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705695 2026-02-20T17:16:29.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-38654705695' 2026-02-20T17:16:29.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:29.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:29.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574877 2026-02-20T17:16:29.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574877 2026-02-20T17:16:29.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-38654705695 2-55834574877' 2026-02-20T17:16:29.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:29.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-20T17:16:29.560 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:29.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:29.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-20T17:16:29.561 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:29.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-20T17:16:29.563 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836514 2026-02-20T17:16:29.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-20T17:16:29.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:29.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836514 2026-02-20T17:16:29.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:30.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:30.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:30.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836514 2026-02-20T17:16:30.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:30.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705695 2026-02-20T17:16:30.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:30.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:30.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705695 2026-02-20T17:16:30.890 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:30.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705695 2026-02-20T17:16:30.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705695' 2026-02-20T17:16:30.892 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705695 2026-02-20T17:16:30.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:31.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705696 -lt 38654705695 2026-02-20T17:16:31.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:31.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574877 2026-02-20T17:16:31.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:31.062 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:31.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574877 2026-02-20T17:16:31.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:31.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574877 2026-02-20T17:16:31.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574877' 2026-02-20T17:16:31.065 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574877 2026-02-20T17:16:31.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:31.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574878 -lt 55834574877 2026-02-20T17:16:31.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:31.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-20T17:16:31.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:31.366 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:31.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-20T17:16:31.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-20T17:16:31.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-20T17:16:31.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-20T17:16:31.377 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set noscrub 2026-02-20T17:16:31.573 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is set 2026-02-20T17:16:31.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-20T17:16:31.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-20T17:16:31.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-20T17:16:31.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:31.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:31.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:31.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:31.744 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:31.745 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:31.745 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:31.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:31.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:31.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:31.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-20T17:16:31.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-20T17:16:31.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-20T17:16:31.816 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:31.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:31.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705699 2026-02-20T17:16:31.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705699 2026-02-20T17:16:31.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705699' 2026-02-20T17:16:31.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:31.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:31.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574880 2026-02-20T17:16:31.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574880 2026-02-20T17:16:31.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705699 2-55834574880' 2026-02-20T17:16:31.958 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:31.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-20T17:16:31.959 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:31.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:31.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-20T17:16:31.961 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:31.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-20T17:16:31.963 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836518 2026-02-20T17:16:31.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-20T17:16:31.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:32.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836518 2026-02-20T17:16:32.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:33.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:33.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:33.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836518 2026-02-20T17:16:33.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:33.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705699 2026-02-20T17:16:33.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:33.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:33.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705699 2026-02-20T17:16:33.289 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:33.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705699 2026-02-20T17:16:33.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705699' 2026-02-20T17:16:33.291 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705699 2026-02-20T17:16:33.291 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:33.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705699 -lt 38654705699 2026-02-20T17:16:33.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:33.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574880 2026-02-20T17:16:33.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:33.459 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:33.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574880 2026-02-20T17:16:33.460 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:33.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574880 2026-02-20T17:16:33.461 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574880 2026-02-20T17:16:33.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574880' 2026-02-20T17:16:33.462 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:33.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574881 -lt 55834574880 2026-02-20T17:16:33.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:33.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:33.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:33.778 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:33.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:33.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:33.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:33.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:33.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:33.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:33.948 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:33.948 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:33.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:33.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:33.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:34.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836521 2026-02-20T17:16:34.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836521 2026-02-20T17:16:34.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521' 2026-02-20T17:16:34.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:34.021 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:34.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705702 2026-02-20T17:16:34.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705702 2026-02-20T17:16:34.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-38654705702' 2026-02-20T17:16:34.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:34.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:34.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574883 2026-02-20T17:16:34.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574883 2026-02-20T17:16:34.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-38654705702 2-55834574883' 2026-02-20T17:16:34.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:34.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836521 2026-02-20T17:16:34.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:34.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:34.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836521 2026-02-20T17:16:34.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:34.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836521 2026-02-20T17:16:34.166 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836521 2026-02-20T17:16:34.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2026-02-20T17:16:34.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:34.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836521 2026-02-20T17:16:34.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:35.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:35.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:35.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836521 -lt 21474836521 2026-02-20T17:16:35.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:35.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705702 2026-02-20T17:16:35.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:35.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705702 2026-02-20T17:16:35.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:35.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705702 2026-02-20T17:16:35.487 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705702 2026-02-20T17:16:35.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705702' 2026-02-20T17:16:35.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:35.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705702 2026-02-20T17:16:35.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:35.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574883 2026-02-20T17:16:35.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:35.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:35.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574883 2026-02-20T17:16:35.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:35.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574883 2026-02-20T17:16:35.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574883' 2026-02-20T17:16:35.659 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574883 2026-02-20T17:16:35.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:35.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574884 -lt 55834574883 2026-02-20T17:16:35.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:35.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:35.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:35.974 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:35.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:35.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:35.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:35.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:35.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:36.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:36.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836524 2026-02-20T17:16:36.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836524 2026-02-20T17:16:36.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836524' 2026-02-20T17:16:36.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:36.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:36.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705705 2026-02-20T17:16:36.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705705 2026-02-20T17:16:36.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836524 1-38654705705' 2026-02-20T17:16:36.290 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:36.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:36.364 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574887 2026-02-20T17:16:36.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574887 2026-02-20T17:16:36.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836524 1-38654705705 2-55834574887' 2026-02-20T17:16:36.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:36.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836524 2026-02-20T17:16:36.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:36.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:36.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836524 2026-02-20T17:16:36.367 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:36.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836524 2026-02-20T17:16:36.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2026-02-20T17:16:36.369 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836524 2026-02-20T17:16:36.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:36.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836524 -lt 21474836524 2026-02-20T17:16:36.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:36.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705705 2026-02-20T17:16:36.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:36.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:36.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705705 2026-02-20T17:16:36.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:36.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705705 2026-02-20T17:16:36.526 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705705 2026-02-20T17:16:36.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705705' 2026-02-20T17:16:36.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:36.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705705 -lt 38654705705 2026-02-20T17:16:36.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:36.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574887 2026-02-20T17:16:36.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:36.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:36.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574887 2026-02-20T17:16:36.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:36.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574887 2026-02-20T17:16:36.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574887' 2026-02-20T17:16:36.701 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574887 2026-02-20T17:16:36.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:36.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574887 -lt 55834574887 2026-02-20T17:16:36.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:36.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:36.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:37.016 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:37.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:37.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:37.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:37.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:37.025 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:37.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836526 2026-02-20T17:16:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836526 2026-02-20T17:16:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526' 2026-02-20T17:16:37.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:37.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:37.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705707 2026-02-20T17:16:37.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705707 2026-02-20T17:16:37.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526 1-38654705707' 2026-02-20T17:16:37.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:37.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574889 2026-02-20T17:16:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574889 2026-02-20T17:16:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526 1-38654705707 2-55834574889' 2026-02-20T17:16:37.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:37.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836526 2026-02-20T17:16:37.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:37.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:37.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836526 2026-02-20T17:16:37.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:37.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836526 2026-02-20T17:16:37.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2026-02-20T17:16:37.399 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836526 2026-02-20T17:16:37.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:37.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836524 -lt 21474836526 2026-02-20T17:16:37.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:38.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:38.569 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:38.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836526 2026-02-20T17:16:38.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:38.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705707 2026-02-20T17:16:38.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:38.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:38.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705707 2026-02-20T17:16:38.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:38.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705707 2026-02-20T17:16:38.732 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705707 2026-02-20T17:16:38.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705707' 2026-02-20T17:16:38.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:38.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705708 -lt 38654705707 2026-02-20T17:16:38.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:38.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574889 2026-02-20T17:16:38.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:38.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:38.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574889 2026-02-20T17:16:38.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:38.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574889 2026-02-20T17:16:38.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574889' 2026-02-20T17:16:38.900 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574889 2026-02-20T17:16:38.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:39.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574890 -lt 55834574889 2026-02-20T17:16:39.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:39.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:39.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:39.216 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:39.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:39.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:39.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:39.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:39.225 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:39.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:39.384 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:39.385 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:39.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:39.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836529 2026-02-20T17:16:39.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836529 2026-02-20T17:16:39.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529' 2026-02-20T17:16:39.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:39.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:39.529 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705710 2026-02-20T17:16:39.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705710 2026-02-20T17:16:39.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705710' 2026-02-20T17:16:39.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:39.530 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:39.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574892 2026-02-20T17:16:39.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574892 2026-02-20T17:16:39.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705710 2-55834574892' 2026-02-20T17:16:39.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:39.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836529 2026-02-20T17:16:39.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:39.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:39.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836529 2026-02-20T17:16:39.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:39.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836529 2026-02-20T17:16:39.603 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836529 2026-02-20T17:16:39.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2026-02-20T17:16:39.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:39.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836529 2026-02-20T17:16:39.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:40.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:40.774 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:40.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836529 2026-02-20T17:16:40.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:40.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705710 2026-02-20T17:16:40.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:40.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:40.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705710 2026-02-20T17:16:40.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:40.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705710 2026-02-20T17:16:40.934 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705710 2026-02-20T17:16:40.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705710' 2026-02-20T17:16:40.935 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:41.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705711 -lt 38654705710 2026-02-20T17:16:41.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:41.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574892 2026-02-20T17:16:41.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:41.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:41.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574892 2026-02-20T17:16:41.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:41.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574892 2026-02-20T17:16:41.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574892' 2026-02-20T17:16:41.098 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574892 2026-02-20T17:16:41.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:41.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574893 -lt 55834574892 2026-02-20T17:16:41.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:41.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:41.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:41.434 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:41.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:41.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:41.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:41.606 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:41.606 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:41.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:41.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:41.606 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:41.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-20T17:16:41.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-20T17:16:41.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-20T17:16:41.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:41.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:41.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705713 2026-02-20T17:16:41.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705713 2026-02-20T17:16:41.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713' 2026-02-20T17:16:41.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:41.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:41.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574895 2026-02-20T17:16:41.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574895 2026-02-20T17:16:41.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713 2-55834574895' 2026-02-20T17:16:41.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:41.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-20T17:16:41.824 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:41.824 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:41.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-20T17:16:41.825 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:41.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-20T17:16:41.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-20T17:16:41.827 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836532 2026-02-20T17:16:41.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:41.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836532 2026-02-20T17:16:41.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:42.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:42.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:43.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836532 2026-02-20T17:16:43.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:43.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705713 2026-02-20T17:16:43.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:43.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:43.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705713 2026-02-20T17:16:43.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:43.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705713 2026-02-20T17:16:43.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705713' 2026-02-20T17:16:43.149 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705713 2026-02-20T17:16:43.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:43.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705714 -lt 38654705713 2026-02-20T17:16:43.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:43.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574895 2026-02-20T17:16:43.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:43.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:43.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574895 2026-02-20T17:16:43.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:43.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574895 2026-02-20T17:16:43.311 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574895 2026-02-20T17:16:43.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574895' 2026-02-20T17:16:43.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:43.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574896 -lt 55834574895 2026-02-20T17:16:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:43.474 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:43.619 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:43.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:43.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:43.628 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:43.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:43.629 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:43.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:43.789 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:43.789 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:43.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:43.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:43.790 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:43.859 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836536 2026-02-20T17:16:43.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836536 2026-02-20T17:16:43.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536' 2026-02-20T17:16:43.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:43.860 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:43.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705717 2026-02-20T17:16:43.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705717 2026-02-20T17:16:43.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-38654705717' 2026-02-20T17:16:43.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:43.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:44.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574898 2026-02-20T17:16:44.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574898 2026-02-20T17:16:44.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-38654705717 2-55834574898' 2026-02-20T17:16:44.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:44.006 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836536 2026-02-20T17:16:44.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:44.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:44.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836536 2026-02-20T17:16:44.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:44.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836536 2026-02-20T17:16:44.010 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836536 2026-02-20T17:16:44.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2026-02-20T17:16:44.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:44.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836536 2026-02-20T17:16:44.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:45.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:45.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:45.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836536 2026-02-20T17:16:45.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:45.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705717 2026-02-20T17:16:45.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:45.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:45.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705717 2026-02-20T17:16:45.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:45.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705717 2026-02-20T17:16:45.329 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705717 2026-02-20T17:16:45.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705717' 2026-02-20T17:16:45.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:45.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705717 -lt 38654705717 2026-02-20T17:16:45.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:45.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574898 2026-02-20T17:16:45.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:45.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:45.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574898 2026-02-20T17:16:45.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:45.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574898 2026-02-20T17:16:45.501 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574898' 2026-02-20T17:16:45.501 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574898 2026-02-20T17:16:45.501 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574899 -lt 55834574898 2026-02-20T17:16:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:45.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:45.810 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:45.821 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:45.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:45.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:45.979 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:46.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836539 2026-02-20T17:16:46.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836539 2026-02-20T17:16:46.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539' 2026-02-20T17:16:46.051 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:46.051 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705720 2026-02-20T17:16:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705720 2026-02-20T17:16:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-38654705720' 2026-02-20T17:16:46.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:46.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:46.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574901 2026-02-20T17:16:46.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574901 2026-02-20T17:16:46.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-38654705720 2-55834574901' 2026-02-20T17:16:46.196 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:46.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836539 2026-02-20T17:16:46.196 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:46.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:46.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836539 2026-02-20T17:16:46.198 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:46.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836539 2026-02-20T17:16:46.200 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836539 2026-02-20T17:16:46.200 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2026-02-20T17:16:46.200 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:46.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836539 2026-02-20T17:16:46.359 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:47.360 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:47.361 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:47.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836539 2026-02-20T17:16:47.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:47.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705720 2026-02-20T17:16:47.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:47.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:47.518 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705720 2026-02-20T17:16:47.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:47.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705720 2026-02-20T17:16:47.520 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705720 2026-02-20T17:16:47.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705720' 2026-02-20T17:16:47.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:47.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705720 -lt 38654705720 2026-02-20T17:16:47.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:47.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574901 2026-02-20T17:16:47.682 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:47.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:47.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574901 2026-02-20T17:16:47.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:47.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574901 2026-02-20T17:16:47.686 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574901 2026-02-20T17:16:47.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574901' 2026-02-20T17:16:47.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:47.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574902 -lt 55834574901 2026-02-20T17:16:47.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:47.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:47.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:47.999 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:48.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:48.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:48.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:48.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:48.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:48.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:48.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836542 2026-02-20T17:16:48.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836542 2026-02-20T17:16:48.237 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542' 2026-02-20T17:16:48.238 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:48.238 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:48.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705723 2026-02-20T17:16:48.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705723 2026-02-20T17:16:48.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542 1-38654705723' 2026-02-20T17:16:48.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:48.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:48.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574905 2026-02-20T17:16:48.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574905 2026-02-20T17:16:48.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836542 1-38654705723 2-55834574905' 2026-02-20T17:16:48.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:48.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836542 2026-02-20T17:16:48.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:48.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:48.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836542 2026-02-20T17:16:48.385 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:48.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836542 2026-02-20T17:16:48.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2026-02-20T17:16:48.386 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836542 2026-02-20T17:16:48.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:48.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836542 -lt 21474836542 2026-02-20T17:16:48.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:48.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705723 2026-02-20T17:16:48.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:48.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:48.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705723 2026-02-20T17:16:48.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:48.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705723 2026-02-20T17:16:48.551 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705723 2026-02-20T17:16:48.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705723' 2026-02-20T17:16:48.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:48.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705723 -lt 38654705723 2026-02-20T17:16:48.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:48.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574905 2026-02-20T17:16:48.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:48.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:48.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574905 2026-02-20T17:16:48.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:48.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574905 2026-02-20T17:16:48.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574905' 2026-02-20T17:16:48.722 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574905 2026-02-20T17:16:48.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:48.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574904 -lt 55834574905 2026-02-20T17:16:48.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:49.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:49.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:50.048 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574904 -lt 55834574905 2026-02-20T17:16:50.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:51.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:16:51.050 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:51.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574907 -lt 55834574905 2026-02-20T17:16:51.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:51.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:51.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:51.360 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:51.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:51.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836546 2026-02-20T17:16:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836546 2026-02-20T17:16:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546' 2026-02-20T17:16:51.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:51.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705727 2026-02-20T17:16:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705727 2026-02-20T17:16:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546 1-38654705727' 2026-02-20T17:16:51.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:51.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:51.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574909 2026-02-20T17:16:51.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574909 2026-02-20T17:16:51.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836546 1-38654705727 2-55834574909' 2026-02-20T17:16:51.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:51.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836546 2026-02-20T17:16:51.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:51.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:51.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836546 2026-02-20T17:16:51.743 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:51.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836546 2026-02-20T17:16:51.745 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2026-02-20T17:16:51.745 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836546 2026-02-20T17:16:51.745 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:51.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836544 -lt 21474836546 2026-02-20T17:16:51.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:52.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:52.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:53.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836546 2026-02-20T17:16:53.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:53.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705727 2026-02-20T17:16:53.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:53.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:53.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705727 2026-02-20T17:16:53.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:53.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705727 2026-02-20T17:16:53.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705727' 2026-02-20T17:16:53.068 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705727 2026-02-20T17:16:53.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705728 -lt 38654705727 2026-02-20T17:16:53.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:53.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574909 2026-02-20T17:16:53.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:53.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:53.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574909 2026-02-20T17:16:53.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:53.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574909 2026-02-20T17:16:53.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574909' 2026-02-20T17:16:53.244 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574909 2026-02-20T17:16:53.244 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:53.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574910 -lt 55834574909 2026-02-20T17:16:53.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:53.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:53.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:53.555 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:53.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:53.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:53.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:53.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:53.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:53.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:53.728 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:53.728 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:53.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:53.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:53.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836550 2026-02-20T17:16:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836550 2026-02-20T17:16:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550' 2026-02-20T17:16:53.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:53.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:53.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705731 2026-02-20T17:16:53.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705731 2026-02-20T17:16:53.870 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731' 2026-02-20T17:16:53.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:53.871 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574912 2026-02-20T17:16:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574912 2026-02-20T17:16:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731 2-55834574912' 2026-02-20T17:16:53.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:53.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836550 2026-02-20T17:16:53.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:53.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:53.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836550 2026-02-20T17:16:53.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:53.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836550 2026-02-20T17:16:53.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2026-02-20T17:16:53.947 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836550 2026-02-20T17:16:53.948 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836550 2026-02-20T17:16:54.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:55.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:55.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:55.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836550 -lt 21474836550 2026-02-20T17:16:55.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:55.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705731 2026-02-20T17:16:55.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:55.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:55.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705731 2026-02-20T17:16:55.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:55.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705731 2026-02-20T17:16:55.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705731' 2026-02-20T17:16:55.278 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705731 2026-02-20T17:16:55.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:55.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705731 -lt 38654705731 2026-02-20T17:16:55.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:55.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574912 2026-02-20T17:16:55.447 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:55.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:55.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574912 2026-02-20T17:16:55.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:55.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574912 2026-02-20T17:16:55.451 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574912 2026-02-20T17:16:55.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574912' 2026-02-20T17:16:55.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:55.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574913 -lt 55834574912 2026-02-20T17:16:55.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:55.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:55.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:55.763 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:55.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:55.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:55.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836553 2026-02-20T17:16:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836553 2026-02-20T17:16:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553' 2026-02-20T17:16:56.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:56.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:56.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705734 2026-02-20T17:16:56.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705734 2026-02-20T17:16:56.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734' 2026-02-20T17:16:56.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:56.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:56.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574915 2026-02-20T17:16:56.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574915 2026-02-20T17:16:56.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734 2-55834574915' 2026-02-20T17:16:56.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:56.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836553 2026-02-20T17:16:56.150 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:56.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:56.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836553 2026-02-20T17:16:56.152 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:56.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836553 2026-02-20T17:16:56.154 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2026-02-20T17:16:56.154 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836553 2026-02-20T17:16:56.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836550 -lt 21474836553 2026-02-20T17:16:56.316 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:16:57.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:16:57.317 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:57.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836553 -lt 21474836553 2026-02-20T17:16:57.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:57.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705734 2026-02-20T17:16:57.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:57.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:57.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705734 2026-02-20T17:16:57.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:57.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705734 2026-02-20T17:16:57.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705734' 2026-02-20T17:16:57.484 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705734 2026-02-20T17:16:57.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:57.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705734 -lt 38654705734 2026-02-20T17:16:57.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:57.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574915 2026-02-20T17:16:57.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:57.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:57.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574915 2026-02-20T17:16:57.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:57.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574915 2026-02-20T17:16:57.651 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574915 2026-02-20T17:16:57.651 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574915' 2026-02-20T17:16:57.651 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:57.813 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574916 -lt 55834574915 2026-02-20T17:16:57.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:57.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:57.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:57.964 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:57.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:57.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:57.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:57.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:57.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:58.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:58.131 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:58.131 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:58.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:58.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:58.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:58.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836556 2026-02-20T17:16:58.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836556 2026-02-20T17:16:58.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836556' 2026-02-20T17:16:58.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:58.207 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:58.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705737 2026-02-20T17:16:58.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705737 2026-02-20T17:16:58.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836556 1-38654705737' 2026-02-20T17:16:58.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:58.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:58.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574919 2026-02-20T17:16:58.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574919 2026-02-20T17:16:58.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836556 1-38654705737 2-55834574919' 2026-02-20T17:16:58.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:58.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836556 2026-02-20T17:16:58.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:58.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:58.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836556 2026-02-20T17:16:58.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:58.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836556 2026-02-20T17:16:58.351 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836556 2026-02-20T17:16:58.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2026-02-20T17:16:58.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:58.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836556 2026-02-20T17:16:58.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:58.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705737 2026-02-20T17:16:58.517 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:58.518 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:16:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705737 2026-02-20T17:16:58.519 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:58.520 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705737 2026-02-20T17:16:58.520 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705737 2026-02-20T17:16:58.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705737' 2026-02-20T17:16:58.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:16:58.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705737 -lt 38654705737 2026-02-20T17:16:58.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:58.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574919 2026-02-20T17:16:58.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:58.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:16:58.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574919 2026-02-20T17:16:58.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:58.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574919 2026-02-20T17:16:58.691 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574919 2026-02-20T17:16:58.691 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574919' 2026-02-20T17:16:58.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:16:58.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574919 -lt 55834574919 2026-02-20T17:16:58.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:16:58.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:16:58.854 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:16:59.005 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:16:59.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:16:59.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:16:59.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:16:59.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:16:59.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:59.174 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:16:59.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836558 2026-02-20T17:16:59.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836558 2026-02-20T17:16:59.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558' 2026-02-20T17:16:59.250 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:59.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:16:59.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705739 2026-02-20T17:16:59.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705739 2026-02-20T17:16:59.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739' 2026-02-20T17:16:59.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:16:59.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:16:59.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574921 2026-02-20T17:16:59.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574921 2026-02-20T17:16:59.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739 2-55834574921' 2026-02-20T17:16:59.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:16:59.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836558 2026-02-20T17:16:59.396 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:16:59.396 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:16:59.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836558 2026-02-20T17:16:59.397 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:16:59.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836558 2026-02-20T17:16:59.399 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836558 2026-02-20T17:16:59.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2026-02-20T17:16:59.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:16:59.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836558 2026-02-20T17:16:59.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:00.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:00.562 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:00.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836558 2026-02-20T17:17:00.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:00.723 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705739 2026-02-20T17:17:00.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:00.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:00.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705739 2026-02-20T17:17:00.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:00.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705739 2026-02-20T17:17:00.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705739' 2026-02-20T17:17:00.727 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705739 2026-02-20T17:17:00.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:00.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705740 -lt 38654705739 2026-02-20T17:17:00.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:00.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574921 2026-02-20T17:17:00.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:00.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:00.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574921 2026-02-20T17:17:00.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574921 2026-02-20T17:17:00.898 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574921 2026-02-20T17:17:00.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574921' 2026-02-20T17:17:00.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:01.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574922 -lt 55834574921 2026-02-20T17:17:01.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:01.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:01.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:01.217 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:01.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:01.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:01.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:01.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:01.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:01.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:01.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836561 2026-02-20T17:17:01.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836561 2026-02-20T17:17:01.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561' 2026-02-20T17:17:01.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:01.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:01.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705742 2026-02-20T17:17:01.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705742 2026-02-20T17:17:01.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742' 2026-02-20T17:17:01.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:01.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:01.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574924 2026-02-20T17:17:01.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574924 2026-02-20T17:17:01.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742 2-55834574924' 2026-02-20T17:17:01.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:01.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836561 2026-02-20T17:17:01.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:01.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:01.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836561 2026-02-20T17:17:01.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:01.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836561 2026-02-20T17:17:01.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2026-02-20T17:17:01.605 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836561 2026-02-20T17:17:01.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836561 2026-02-20T17:17:01.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:02.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:02.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:02.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836561 2026-02-20T17:17:02.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:02.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705742 2026-02-20T17:17:02.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:02.928 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:02.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705742 2026-02-20T17:17:02.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:02.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705742 2026-02-20T17:17:02.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705742' 2026-02-20T17:17:02.931 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705742 2026-02-20T17:17:02.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:03.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705743 -lt 38654705742 2026-02-20T17:17:03.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:03.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574924 2026-02-20T17:17:03.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:03.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:03.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574924 2026-02-20T17:17:03.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:03.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574924 2026-02-20T17:17:03.102 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574924 2026-02-20T17:17:03.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574924' 2026-02-20T17:17:03.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:03.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574925 -lt 55834574924 2026-02-20T17:17:03.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:03.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:03.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:03.419 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:03.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:03.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:03.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:03.581 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:03.581 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:03.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:03.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:03.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:03.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836564 2026-02-20T17:17:03.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836564 2026-02-20T17:17:03.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564' 2026-02-20T17:17:03.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:03.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:03.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705745 2026-02-20T17:17:03.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705745 2026-02-20T17:17:03.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705745' 2026-02-20T17:17:03.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:03.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:03.802 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574927 2026-02-20T17:17:03.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574927 2026-02-20T17:17:03.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705745 2-55834574927' 2026-02-20T17:17:03.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:03.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836564 2026-02-20T17:17:03.803 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:03.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:03.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836564 2026-02-20T17:17:03.805 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:03.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836564 2026-02-20T17:17:03.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2026-02-20T17:17:03.807 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836564 2026-02-20T17:17:03.807 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:03.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836564 2026-02-20T17:17:03.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:04.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:04.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:05.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836564 2026-02-20T17:17:05.137 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:05.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705745 2026-02-20T17:17:05.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:05.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:05.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705745 2026-02-20T17:17:05.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:05.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705745 2026-02-20T17:17:05.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705745' 2026-02-20T17:17:05.141 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705745 2026-02-20T17:17:05.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:05.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705746 -lt 38654705745 2026-02-20T17:17:05.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:05.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574927 2026-02-20T17:17:05.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:05.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:05.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574927 2026-02-20T17:17:05.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:05.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574927 2026-02-20T17:17:05.311 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574927 2026-02-20T17:17:05.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574927' 2026-02-20T17:17:05.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:05.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574928 -lt 55834574927 2026-02-20T17:17:05.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:05.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:05.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:05.630 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:05.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:05.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:05.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:05.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:05.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:05.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:05.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836568 2026-02-20T17:17:05.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836568 2026-02-20T17:17:05.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568' 2026-02-20T17:17:05.873 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:05.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705749 2026-02-20T17:17:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705749 2026-02-20T17:17:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568 1-38654705749' 2026-02-20T17:17:05.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:05.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:06.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574930 2026-02-20T17:17:06.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574930 2026-02-20T17:17:06.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568 1-38654705749 2-55834574930' 2026-02-20T17:17:06.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:06.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836568 2026-02-20T17:17:06.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:06.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:06.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836568 2026-02-20T17:17:06.018 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:06.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836568 2026-02-20T17:17:06.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2026-02-20T17:17:06.019 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836568 2026-02-20T17:17:06.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:06.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836568 2026-02-20T17:17:06.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:07.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:07.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:07.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836568 2026-02-20T17:17:07.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:07.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705749 2026-02-20T17:17:07.347 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:07.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:07.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705749 2026-02-20T17:17:07.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:07.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705749 2026-02-20T17:17:07.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705749' 2026-02-20T17:17:07.351 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705749 2026-02-20T17:17:07.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:07.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705749 -lt 38654705749 2026-02-20T17:17:07.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:07.521 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574930 2026-02-20T17:17:07.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:07.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:07.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574930 2026-02-20T17:17:07.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:07.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574930 2026-02-20T17:17:07.525 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574930 2026-02-20T17:17:07.525 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574930' 2026-02-20T17:17:07.526 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:07.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574931 -lt 55834574930 2026-02-20T17:17:07.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:07.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:07.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:07.831 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:07.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:07.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:07.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:07.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:07.841 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:07.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:08.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836571 2026-02-20T17:17:08.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836571 2026-02-20T17:17:08.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571' 2026-02-20T17:17:08.072 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:08.072 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:08.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705752 2026-02-20T17:17:08.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705752 2026-02-20T17:17:08.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571 1-38654705752' 2026-02-20T17:17:08.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:08.146 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:08.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574933 2026-02-20T17:17:08.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574933 2026-02-20T17:17:08.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571 1-38654705752 2-55834574933' 2026-02-20T17:17:08.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:08.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836571 2026-02-20T17:17:08.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:08.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:08.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836571 2026-02-20T17:17:08.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:08.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836571 2026-02-20T17:17:08.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2026-02-20T17:17:08.223 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836571 2026-02-20T17:17:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836571 2026-02-20T17:17:08.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:09.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:09.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:09.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836571 2026-02-20T17:17:09.548 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:09.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705752 2026-02-20T17:17:09.549 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:09.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:09.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705752 2026-02-20T17:17:09.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:09.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705752 2026-02-20T17:17:09.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705752' 2026-02-20T17:17:09.552 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705752 2026-02-20T17:17:09.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705752 -lt 38654705752 2026-02-20T17:17:09.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:09.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574933 2026-02-20T17:17:09.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:09.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:09.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574933 2026-02-20T17:17:09.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:09.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574933 2026-02-20T17:17:09.724 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574933' 2026-02-20T17:17:09.724 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574933 2026-02-20T17:17:09.724 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:09.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574934 -lt 55834574933 2026-02-20T17:17:09.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:09.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:09.888 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:10.033 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:10.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:10.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:10.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:10.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:10.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:10.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:10.209 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:10.209 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:10.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:10.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:10.210 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:10.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836574 2026-02-20T17:17:10.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836574 2026-02-20T17:17:10.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836574' 2026-02-20T17:17:10.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:10.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:10.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705755 2026-02-20T17:17:10.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705755 2026-02-20T17:17:10.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836574 1-38654705755' 2026-02-20T17:17:10.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:10.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:10.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574937 2026-02-20T17:17:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574937 2026-02-20T17:17:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836574 1-38654705755 2-55834574937' 2026-02-20T17:17:10.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:10.430 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836574 2026-02-20T17:17:10.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:10.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:10.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836574 2026-02-20T17:17:10.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:10.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836574 2026-02-20T17:17:10.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2026-02-20T17:17:10.434 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836574 2026-02-20T17:17:10.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:10.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836574 -lt 21474836574 2026-02-20T17:17:10.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:10.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705755 2026-02-20T17:17:10.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:10.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:10.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705755 2026-02-20T17:17:10.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:10.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705755 2026-02-20T17:17:10.603 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705755 2026-02-20T17:17:10.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705755' 2026-02-20T17:17:10.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705755 -lt 38654705755 2026-02-20T17:17:10.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:10.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574937 2026-02-20T17:17:10.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:10.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:10.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574937 2026-02-20T17:17:10.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:10.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574937 2026-02-20T17:17:10.771 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574937 2026-02-20T17:17:10.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574937' 2026-02-20T17:17:10.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:10.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574936 -lt 55834574937 2026-02-20T17:17:10.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:11.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:11.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:12.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574936 -lt 55834574937 2026-02-20T17:17:12.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:13.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:17:13.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:13.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574939 -lt 55834574937 2026-02-20T17:17:13.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:13.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:13.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:13.407 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:13.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:13.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:13.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:13.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:13.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:13.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836578 2026-02-20T17:17:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836578 2026-02-20T17:17:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578' 2026-02-20T17:17:13.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:13.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:13.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705759 2026-02-20T17:17:13.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705759 2026-02-20T17:17:13.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-38654705759' 2026-02-20T17:17:13.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:13.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:13.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574941 2026-02-20T17:17:13.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574941 2026-02-20T17:17:13.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-38654705759 2-55834574941' 2026-02-20T17:17:13.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:13.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836578 2026-02-20T17:17:13.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:13.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:13.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836578 2026-02-20T17:17:13.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:13.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836578 2026-02-20T17:17:13.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2026-02-20T17:17:13.795 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836578 2026-02-20T17:17:13.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:13.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836576 -lt 21474836578 2026-02-20T17:17:13.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:14.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:14.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836578 2026-02-20T17:17:15.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:15.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705759 2026-02-20T17:17:15.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:15.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:15.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705759 2026-02-20T17:17:15.119 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705759 2026-02-20T17:17:15.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705759' 2026-02-20T17:17:15.120 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705759 2026-02-20T17:17:15.121 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:15.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705760 -lt 38654705759 2026-02-20T17:17:15.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:15.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574941 2026-02-20T17:17:15.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:15.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:15.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574941 2026-02-20T17:17:15.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:15.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574941 2026-02-20T17:17:15.278 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574941 2026-02-20T17:17:15.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574941' 2026-02-20T17:17:15.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:15.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574942 -lt 55834574941 2026-02-20T17:17:15.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:15.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:15.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:15.592 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:15.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:15.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:15.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:15.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:15.764 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:15.764 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:15.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:15.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:15.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:15.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836582 2026-02-20T17:17:15.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836582 2026-02-20T17:17:15.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582' 2026-02-20T17:17:15.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:15.835 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705763 2026-02-20T17:17:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705763 2026-02-20T17:17:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705763' 2026-02-20T17:17:15.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:15.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574944 2026-02-20T17:17:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574944 2026-02-20T17:17:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705763 2-55834574944' 2026-02-20T17:17:15.980 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:15.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836582 2026-02-20T17:17:15.981 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:15.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:15.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836582 2026-02-20T17:17:15.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:15.983 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836582 2026-02-20T17:17:15.984 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836582 2026-02-20T17:17:15.984 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2026-02-20T17:17:15.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:16.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836582 2026-02-20T17:17:16.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:17.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:17.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:17.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836582 -lt 21474836582 2026-02-20T17:17:17.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:17.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705763 2026-02-20T17:17:17.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:17.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:17.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705763 2026-02-20T17:17:17.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:17.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705763 2026-02-20T17:17:17.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705763' 2026-02-20T17:17:17.305 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705763 2026-02-20T17:17:17.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:17.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705763 -lt 38654705763 2026-02-20T17:17:17.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:17.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574944 2026-02-20T17:17:17.466 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:17.467 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:17.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574944 2026-02-20T17:17:17.468 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:17.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574944 2026-02-20T17:17:17.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574944' 2026-02-20T17:17:17.470 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574944 2026-02-20T17:17:17.470 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:17.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574945 -lt 55834574944 2026-02-20T17:17:17.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:17.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:17.627 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:17.771 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:17.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:17.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:17.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:17.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:17.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:17.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:17.941 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:17.941 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:17.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:17.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:17.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:18.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836585 2026-02-20T17:17:18.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836585 2026-02-20T17:17:18.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585' 2026-02-20T17:17:18.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:18.017 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:18.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705766 2026-02-20T17:17:18.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705766 2026-02-20T17:17:18.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705766' 2026-02-20T17:17:18.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:18.090 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:18.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574947 2026-02-20T17:17:18.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574947 2026-02-20T17:17:18.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705766 2-55834574947' 2026-02-20T17:17:18.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:18.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836585 2026-02-20T17:17:18.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:18.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:18.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836585 2026-02-20T17:17:18.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:18.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836585 2026-02-20T17:17:18.167 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836585 2026-02-20T17:17:18.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2026-02-20T17:17:18.167 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:18.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836582 -lt 21474836585 2026-02-20T17:17:18.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:19.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:19.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:19.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836585 -lt 21474836585 2026-02-20T17:17:19.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:19.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705766 2026-02-20T17:17:19.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:19.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:19.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705766 2026-02-20T17:17:19.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:19.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705766 2026-02-20T17:17:19.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705766' 2026-02-20T17:17:19.490 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705766 2026-02-20T17:17:19.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:19.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705766 -lt 38654705766 2026-02-20T17:17:19.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:19.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574947 2026-02-20T17:17:19.657 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:19.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:19.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574947 2026-02-20T17:17:19.659 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:19.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574947 2026-02-20T17:17:19.661 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574947 2026-02-20T17:17:19.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574947' 2026-02-20T17:17:19.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:19.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574948 -lt 55834574947 2026-02-20T17:17:19.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:19.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:19.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:19.967 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:19.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:19.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:20.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:20.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836588 2026-02-20T17:17:20.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836588 2026-02-20T17:17:20.208 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588' 2026-02-20T17:17:20.209 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:20.209 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:20.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705769 2026-02-20T17:17:20.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705769 2026-02-20T17:17:20.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-38654705769' 2026-02-20T17:17:20.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:20.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:20.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574951 2026-02-20T17:17:20.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574951 2026-02-20T17:17:20.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-38654705769 2-55834574951' 2026-02-20T17:17:20.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:20.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836588 2026-02-20T17:17:20.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:20.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:20.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836588 2026-02-20T17:17:20.357 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:20.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836588 2026-02-20T17:17:20.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2026-02-20T17:17:20.359 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836588 2026-02-20T17:17:20.359 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:20.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836588 -lt 21474836588 2026-02-20T17:17:20.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:20.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705769 2026-02-20T17:17:20.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:20.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:20.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705769 2026-02-20T17:17:20.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:20.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705769 2026-02-20T17:17:20.527 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705769 2026-02-20T17:17:20.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705769' 2026-02-20T17:17:20.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705769 -lt 38654705769 2026-02-20T17:17:20.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:20.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574951 2026-02-20T17:17:20.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:20.697 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:20.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574951 2026-02-20T17:17:20.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:20.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574951 2026-02-20T17:17:20.699 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574951 2026-02-20T17:17:20.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574951' 2026-02-20T17:17:20.700 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:20.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574951 -lt 55834574951 2026-02-20T17:17:20.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:20.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:20.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:21.014 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:21.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:21.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:21.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:21.024 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:21.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:21.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:21.181 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:21.181 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:21.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:21.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:21.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:21.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836590 2026-02-20T17:17:21.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836590 2026-02-20T17:17:21.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590' 2026-02-20T17:17:21.256 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:21.256 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:21.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705771 2026-02-20T17:17:21.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705771 2026-02-20T17:17:21.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771' 2026-02-20T17:17:21.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:21.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:21.397 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574953 2026-02-20T17:17:21.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574953 2026-02-20T17:17:21.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771 2-55834574953' 2026-02-20T17:17:21.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:21.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836590 2026-02-20T17:17:21.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:21.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:21.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836590 2026-02-20T17:17:21.400 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:21.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836590 2026-02-20T17:17:21.402 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2026-02-20T17:17:21.402 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836590 2026-02-20T17:17:21.402 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:21.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836588 -lt 21474836590 2026-02-20T17:17:21.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:22.569 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:22.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:22.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836591 -lt 21474836590 2026-02-20T17:17:22.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:22.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705771 2026-02-20T17:17:22.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:22.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:22.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705771 2026-02-20T17:17:22.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:22.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705771 2026-02-20T17:17:22.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705771' 2026-02-20T17:17:22.730 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705771 2026-02-20T17:17:22.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:22.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705772 -lt 38654705771 2026-02-20T17:17:22.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:22.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574953 2026-02-20T17:17:22.898 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:22.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:22.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574953 2026-02-20T17:17:22.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:22.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574953 2026-02-20T17:17:22.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574953' 2026-02-20T17:17:22.902 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574953 2026-02-20T17:17:22.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:23.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574954 -lt 55834574953 2026-02-20T17:17:23.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:23.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:23.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:23.211 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:23.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:23.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:23.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:23.384 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:23.384 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:23.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:23.384 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:23.384 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:23.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836593 2026-02-20T17:17:23.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836593 2026-02-20T17:17:23.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593' 2026-02-20T17:17:23.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:23.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:23.527 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705774 2026-02-20T17:17:23.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705774 2026-02-20T17:17:23.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774' 2026-02-20T17:17:23.528 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:23.528 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574956 2026-02-20T17:17:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574956 2026-02-20T17:17:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774 2-55834574956' 2026-02-20T17:17:23.598 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:23.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836593 2026-02-20T17:17:23.599 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:23.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:23.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836593 2026-02-20T17:17:23.601 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:23.602 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836593 2026-02-20T17:17:23.602 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836593 2026-02-20T17:17:23.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2026-02-20T17:17:23.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:23.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836591 -lt 21474836593 2026-02-20T17:17:23.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:24.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:24.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836594 -lt 21474836593 2026-02-20T17:17:24.927 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:24.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705774 2026-02-20T17:17:24.928 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:24.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:24.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705774 2026-02-20T17:17:24.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:24.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705774 2026-02-20T17:17:24.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705774' 2026-02-20T17:17:24.931 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705774 2026-02-20T17:17:24.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:25.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705775 -lt 38654705774 2026-02-20T17:17:25.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:25.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574956 2026-02-20T17:17:25.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:25.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:25.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574956 2026-02-20T17:17:25.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:25.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574956 2026-02-20T17:17:25.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574956' 2026-02-20T17:17:25.097 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574956 2026-02-20T17:17:25.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:25.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574957 -lt 55834574956 2026-02-20T17:17:25.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:25.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:25.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:25.406 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:25.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:25.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:25.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:25.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:25.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:25.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:25.577 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:25.577 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:25.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:25.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:25.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:25.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836596 2026-02-20T17:17:25.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836596 2026-02-20T17:17:25.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596' 2026-02-20T17:17:25.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:25.648 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:25.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705777 2026-02-20T17:17:25.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705777 2026-02-20T17:17:25.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777' 2026-02-20T17:17:25.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:25.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574959 2026-02-20T17:17:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574959 2026-02-20T17:17:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777 2-55834574959' 2026-02-20T17:17:25.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:25.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836596 2026-02-20T17:17:25.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:25.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:25.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836596 2026-02-20T17:17:25.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:25.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836596 2026-02-20T17:17:25.797 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836596 2026-02-20T17:17:25.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2026-02-20T17:17:25.798 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:25.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836594 -lt 21474836596 2026-02-20T17:17:25.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:26.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:26.956 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836597 -lt 21474836596 2026-02-20T17:17:27.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:27.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705777 2026-02-20T17:17:27.116 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:27.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:27.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705777 2026-02-20T17:17:27.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:27.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705777 2026-02-20T17:17:27.119 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705777 2026-02-20T17:17:27.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705777' 2026-02-20T17:17:27.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:27.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705778 -lt 38654705777 2026-02-20T17:17:27.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:27.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574959 2026-02-20T17:17:27.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:27.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:27.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574959 2026-02-20T17:17:27.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:27.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574959 2026-02-20T17:17:27.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574959' 2026-02-20T17:17:27.288 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574959 2026-02-20T17:17:27.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:27.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574960 -lt 55834574959 2026-02-20T17:17:27.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:27.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:27.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:27.608 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:27.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:27.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:27.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:27.618 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:27.618 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:27.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:27.776 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:27.776 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:27.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:27.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:27.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:27.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836600 2026-02-20T17:17:27.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836600 2026-02-20T17:17:27.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600' 2026-02-20T17:17:27.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:27.847 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:27.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705781 2026-02-20T17:17:27.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705781 2026-02-20T17:17:27.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600 1-38654705781' 2026-02-20T17:17:27.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:27.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:27.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574962 2026-02-20T17:17:27.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574962 2026-02-20T17:17:27.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600 1-38654705781 2-55834574962' 2026-02-20T17:17:27.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:27.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836600 2026-02-20T17:17:27.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:27.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:27.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836600 2026-02-20T17:17:27.993 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:27.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836600 2026-02-20T17:17:27.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2026-02-20T17:17:27.995 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836600 2026-02-20T17:17:27.995 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:28.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836597 -lt 21474836600 2026-02-20T17:17:28.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:29.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:29.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:29.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836600 -lt 21474836600 2026-02-20T17:17:29.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:29.323 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705781 2026-02-20T17:17:29.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:29.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:29.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705781 2026-02-20T17:17:29.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:29.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705781 2026-02-20T17:17:29.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705781' 2026-02-20T17:17:29.327 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705781 2026-02-20T17:17:29.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:29.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705781 -lt 38654705781 2026-02-20T17:17:29.488 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:29.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574962 2026-02-20T17:17:29.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:29.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:29.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574962 2026-02-20T17:17:29.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:29.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574962 2026-02-20T17:17:29.492 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574962 2026-02-20T17:17:29.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574962' 2026-02-20T17:17:29.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:29.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574963 -lt 55834574962 2026-02-20T17:17:29.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:29.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:29.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:29.809 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:29.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:29.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:29.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:29.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:29.819 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:29.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:30.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836603 2026-02-20T17:17:30.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836603 2026-02-20T17:17:30.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603' 2026-02-20T17:17:30.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:30.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:30.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705784 2026-02-20T17:17:30.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705784 2026-02-20T17:17:30.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603 1-38654705784' 2026-02-20T17:17:30.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:30.118 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574965 2026-02-20T17:17:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574965 2026-02-20T17:17:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603 1-38654705784 2-55834574965' 2026-02-20T17:17:30.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:30.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836603 2026-02-20T17:17:30.191 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:30.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:30.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836603 2026-02-20T17:17:30.193 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:30.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836603 2026-02-20T17:17:30.195 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2026-02-20T17:17:30.195 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836603 2026-02-20T17:17:30.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:30.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836600 -lt 21474836603 2026-02-20T17:17:30.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:31.365 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:31.365 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:31.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836603 -lt 21474836603 2026-02-20T17:17:31.519 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:31.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705784 2026-02-20T17:17:31.520 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:31.521 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:31.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705784 2026-02-20T17:17:31.522 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705784 2026-02-20T17:17:31.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705784' 2026-02-20T17:17:31.524 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705784 2026-02-20T17:17:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:31.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705784 -lt 38654705784 2026-02-20T17:17:31.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:31.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574965 2026-02-20T17:17:31.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:31.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574965 2026-02-20T17:17:31.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:31.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574965 2026-02-20T17:17:31.689 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574965 2026-02-20T17:17:31.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574965' 2026-02-20T17:17:31.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:31.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574966 -lt 55834574965 2026-02-20T17:17:31.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:31.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:31.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:31.991 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:32.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:32.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:32.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:32.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:32.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:32.165 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:32.165 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:32.166 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:32.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:32.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:32.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:32.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836606 2026-02-20T17:17:32.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836606 2026-02-20T17:17:32.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836606' 2026-02-20T17:17:32.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:32.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:32.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705787 2026-02-20T17:17:32.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705787 2026-02-20T17:17:32.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836606 1-38654705787' 2026-02-20T17:17:32.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:32.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:32.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574969 2026-02-20T17:17:32.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574969 2026-02-20T17:17:32.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836606 1-38654705787 2-55834574969' 2026-02-20T17:17:32.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:32.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836606 2026-02-20T17:17:32.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:32.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:32.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836606 2026-02-20T17:17:32.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:32.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836606 2026-02-20T17:17:32.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836606' 2026-02-20T17:17:32.393 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836606 2026-02-20T17:17:32.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:32.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836606 -lt 21474836606 2026-02-20T17:17:32.554 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:32.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705787 2026-02-20T17:17:32.555 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:32.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:32.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705787 2026-02-20T17:17:32.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:32.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705787 2026-02-20T17:17:32.559 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705787 2026-02-20T17:17:32.559 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705787' 2026-02-20T17:17:32.559 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705787 -lt 38654705787 2026-02-20T17:17:32.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:32.725 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574969 2026-02-20T17:17:32.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:32.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:32.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574969 2026-02-20T17:17:32.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:32.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574969 2026-02-20T17:17:32.729 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574969 2026-02-20T17:17:32.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574969' 2026-02-20T17:17:32.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:32.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574968 -lt 55834574969 2026-02-20T17:17:32.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:33.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:33.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:34.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574968 -lt 55834574969 2026-02-20T17:17:34.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:35.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:17:35.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:35.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574971 -lt 55834574969 2026-02-20T17:17:35.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:35.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:35.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:35.372 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:35.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:35.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:35.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:35.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:35.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:35.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:35.539 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:35.539 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:35.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:35.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:35.539 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:35.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836610 2026-02-20T17:17:35.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836610 2026-02-20T17:17:35.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610' 2026-02-20T17:17:35.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:35.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:35.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705791 2026-02-20T17:17:35.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705791 2026-02-20T17:17:35.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791' 2026-02-20T17:17:35.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:35.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:35.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574973 2026-02-20T17:17:35.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574973 2026-02-20T17:17:35.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791 2-55834574973' 2026-02-20T17:17:35.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:35.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836610 2026-02-20T17:17:35.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:35.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:35.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836610 2026-02-20T17:17:35.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:35.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836610 2026-02-20T17:17:35.759 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2026-02-20T17:17:35.759 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836610 2026-02-20T17:17:35.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:35.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836608 -lt 21474836610 2026-02-20T17:17:35.929 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:36.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:36.931 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836611 -lt 21474836610 2026-02-20T17:17:37.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:37.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705791 2026-02-20T17:17:37.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:37.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:37.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705791 2026-02-20T17:17:37.094 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:37.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705791 2026-02-20T17:17:37.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705791' 2026-02-20T17:17:37.095 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705791 2026-02-20T17:17:37.095 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:37.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705792 -lt 38654705791 2026-02-20T17:17:37.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:37.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574973 2026-02-20T17:17:37.261 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:37.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:37.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574973 2026-02-20T17:17:37.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:37.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574973 2026-02-20T17:17:37.264 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574973 2026-02-20T17:17:37.264 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574973' 2026-02-20T17:17:37.265 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574974 -lt 55834574973 2026-02-20T17:17:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:37.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:37.573 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:37.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:37.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:37.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:37.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:37.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:37.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:37.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836614 2026-02-20T17:17:37.807 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836614 2026-02-20T17:17:37.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614' 2026-02-20T17:17:37.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:37.808 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:37.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705795 2026-02-20T17:17:37.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705795 2026-02-20T17:17:37.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705795' 2026-02-20T17:17:37.878 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:37.878 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:37.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574976 2026-02-20T17:17:37.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574976 2026-02-20T17:17:37.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705795 2-55834574976' 2026-02-20T17:17:37.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:37.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836614 2026-02-20T17:17:37.951 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:37.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:37.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836614 2026-02-20T17:17:37.953 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:37.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836614 2026-02-20T17:17:37.954 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836614 2026-02-20T17:17:37.954 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2026-02-20T17:17:37.955 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:38.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836611 -lt 21474836614 2026-02-20T17:17:38.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:39.112 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:39.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:39.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836614 -lt 21474836614 2026-02-20T17:17:39.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:39.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705795 2026-02-20T17:17:39.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:39.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:39.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705795 2026-02-20T17:17:39.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:39.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705795 2026-02-20T17:17:39.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705795' 2026-02-20T17:17:39.272 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705795 2026-02-20T17:17:39.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:39.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705795 -lt 38654705795 2026-02-20T17:17:39.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:39.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574976 2026-02-20T17:17:39.432 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:39.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:39.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574976 2026-02-20T17:17:39.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:39.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574976 2026-02-20T17:17:39.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574976' 2026-02-20T17:17:39.436 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574976 2026-02-20T17:17:39.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:39.605 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574977 -lt 55834574976 2026-02-20T17:17:39.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:39.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:39.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:39.751 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:39.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:39.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:39.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:39.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:39.760 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:39.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:39.916 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:39.916 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:39.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:39.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:39.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:39.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836617 2026-02-20T17:17:39.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836617 2026-02-20T17:17:39.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617' 2026-02-20T17:17:39.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:39.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:40.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705798 2026-02-20T17:17:40.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705798 2026-02-20T17:17:40.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705798' 2026-02-20T17:17:40.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:40.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:40.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574979 2026-02-20T17:17:40.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574979 2026-02-20T17:17:40.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705798 2-55834574979' 2026-02-20T17:17:40.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:40.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836617 2026-02-20T17:17:40.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:40.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:40.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836617 2026-02-20T17:17:40.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:40.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836617 2026-02-20T17:17:40.135 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836617' 2026-02-20T17:17:40.136 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836617 2026-02-20T17:17:40.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:40.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836614 -lt 21474836617 2026-02-20T17:17:40.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:41.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:41.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:41.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836617 -lt 21474836617 2026-02-20T17:17:41.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:41.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705798 2026-02-20T17:17:41.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:41.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:41.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705798 2026-02-20T17:17:41.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:41.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705798 2026-02-20T17:17:41.463 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705798' 2026-02-20T17:17:41.463 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705798 2026-02-20T17:17:41.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:41.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705798 -lt 38654705798 2026-02-20T17:17:41.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:41.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574979 2026-02-20T17:17:41.633 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:41.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:41.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574979 2026-02-20T17:17:41.635 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:41.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574979 2026-02-20T17:17:41.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574979' 2026-02-20T17:17:41.637 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574979 2026-02-20T17:17:41.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:41.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574980 -lt 55834574979 2026-02-20T17:17:41.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:41.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:41.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:41.940 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:41.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:41.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:41.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:41.950 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:41.950 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:42.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:42.105 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:42.105 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:42.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:42.105 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:42.106 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:42.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836620 2026-02-20T17:17:42.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836620 2026-02-20T17:17:42.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620' 2026-02-20T17:17:42.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:42.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:42.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705801 2026-02-20T17:17:42.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705801 2026-02-20T17:17:42.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801' 2026-02-20T17:17:42.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:42.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574982 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574982 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801 2-55834574982' 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836620 2026-02-20T17:17:42.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:42.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:42.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836620 2026-02-20T17:17:42.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:42.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836620 2026-02-20T17:17:42.329 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836620 2026-02-20T17:17:42.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836620' 2026-02-20T17:17:42.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:42.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836620 -lt 21474836620 2026-02-20T17:17:42.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705801 2026-02-20T17:17:42.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:42.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:42.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705801 2026-02-20T17:17:42.495 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:42.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705801 2026-02-20T17:17:42.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705801' 2026-02-20T17:17:42.496 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705801 2026-02-20T17:17:42.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:42.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705801 -lt 38654705801 2026-02-20T17:17:42.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:42.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574982 2026-02-20T17:17:42.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:42.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:42.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574982 2026-02-20T17:17:42.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:42.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574982 2026-02-20T17:17:42.658 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574982 2026-02-20T17:17:42.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574982' 2026-02-20T17:17:42.658 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:42.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574983 -lt 55834574982 2026-02-20T17:17:42.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:42.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:42.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:42.965 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:42.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:42.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:42.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:42.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:42.975 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:43.131 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:43.132 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:43.132 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:43.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:43.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:43.132 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:43.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836622 2026-02-20T17:17:43.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836622 2026-02-20T17:17:43.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622' 2026-02-20T17:17:43.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:43.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:43.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705803 2026-02-20T17:17:43.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705803 2026-02-20T17:17:43.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622 1-38654705803' 2026-02-20T17:17:43.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:43.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:43.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574985 2026-02-20T17:17:43.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574985 2026-02-20T17:17:43.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836622 1-38654705803 2-55834574985' 2026-02-20T17:17:43.348 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:43.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836622 2026-02-20T17:17:43.349 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:43.350 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:43.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836622 2026-02-20T17:17:43.351 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:43.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836622 2026-02-20T17:17:43.352 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836622 2026-02-20T17:17:43.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836622' 2026-02-20T17:17:43.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836620 -lt 21474836622 2026-02-20T17:17:43.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:44.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:44.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:44.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836623 -lt 21474836622 2026-02-20T17:17:44.672 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:44.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705803 2026-02-20T17:17:44.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:44.674 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:44.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705803 2026-02-20T17:17:44.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:44.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705803 2026-02-20T17:17:44.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705803' 2026-02-20T17:17:44.678 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705803 2026-02-20T17:17:44.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:44.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705804 -lt 38654705803 2026-02-20T17:17:44.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:44.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574985 2026-02-20T17:17:44.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:44.839 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:44.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574985 2026-02-20T17:17:44.839 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574985 2026-02-20T17:17:44.841 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574985' 2026-02-20T17:17:44.842 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574985 2026-02-20T17:17:44.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:45.003 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574986 -lt 55834574985 2026-02-20T17:17:45.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:45.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:45.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:45.150 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:45.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:45.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:45.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:45.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:45.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:45.314 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:45.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836625 2026-02-20T17:17:45.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836625 2026-02-20T17:17:45.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625' 2026-02-20T17:17:45.387 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:45.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:45.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705806 2026-02-20T17:17:45.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705806 2026-02-20T17:17:45.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625 1-38654705806' 2026-02-20T17:17:45.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:45.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:45.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574988 2026-02-20T17:17:45.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574988 2026-02-20T17:17:45.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836625 1-38654705806 2-55834574988' 2026-02-20T17:17:45.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:45.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836625 2026-02-20T17:17:45.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:45.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:45.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836625 2026-02-20T17:17:45.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:45.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836625 2026-02-20T17:17:45.535 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836625 2026-02-20T17:17:45.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836625' 2026-02-20T17:17:45.535 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:45.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836623 -lt 21474836625 2026-02-20T17:17:45.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:46.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:46.702 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:46.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836626 -lt 21474836625 2026-02-20T17:17:46.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:46.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705806 2026-02-20T17:17:46.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:46.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:46.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705806 2026-02-20T17:17:46.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:46.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705806 2026-02-20T17:17:46.867 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705806 2026-02-20T17:17:46.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705806' 2026-02-20T17:17:46.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:47.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705807 -lt 38654705806 2026-02-20T17:17:47.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:47.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574988 2026-02-20T17:17:47.036 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:47.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:47.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574988 2026-02-20T17:17:47.038 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:47.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574988 2026-02-20T17:17:47.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574988' 2026-02-20T17:17:47.040 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574988 2026-02-20T17:17:47.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:47.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574989 -lt 55834574988 2026-02-20T17:17:47.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:47.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:47.207 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:47.353 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:47.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:47.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:47.362 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:47.363 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:47.363 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:47.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:47.514 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:47.514 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:47.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:47.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:47.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:47.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836628 2026-02-20T17:17:47.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836628 2026-02-20T17:17:47.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628' 2026-02-20T17:17:47.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:47.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:47.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705809 2026-02-20T17:17:47.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705809 2026-02-20T17:17:47.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628 1-38654705809' 2026-02-20T17:17:47.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:47.661 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:47.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574991 2026-02-20T17:17:47.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574991 2026-02-20T17:17:47.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836628 1-38654705809 2-55834574991' 2026-02-20T17:17:47.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:47.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836628 2026-02-20T17:17:47.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:47.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:47.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836628 2026-02-20T17:17:47.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:47.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836628 2026-02-20T17:17:47.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836628' 2026-02-20T17:17:47.735 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836628 2026-02-20T17:17:47.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:47.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836626 -lt 21474836628 2026-02-20T17:17:47.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:48.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:48.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:49.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836628 2026-02-20T17:17:49.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:49.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705809 2026-02-20T17:17:49.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:49.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:49.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705809 2026-02-20T17:17:49.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:49.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705809 2026-02-20T17:17:49.070 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705809 2026-02-20T17:17:49.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705809' 2026-02-20T17:17:49.070 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:49.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705810 -lt 38654705809 2026-02-20T17:17:49.239 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:49.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574991 2026-02-20T17:17:49.240 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:49.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:49.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574991 2026-02-20T17:17:49.242 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:49.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574991 2026-02-20T17:17:49.243 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574991 2026-02-20T17:17:49.243 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574991' 2026-02-20T17:17:49.243 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574992 -lt 55834574991 2026-02-20T17:17:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:49.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:49.558 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:49.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:49.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:49.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:49.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:49.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:49.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:49.720 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:49.720 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:49.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:49.720 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:49.721 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:49.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836631 2026-02-20T17:17:49.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836631 2026-02-20T17:17:49.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631' 2026-02-20T17:17:49.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:49.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:49.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705813 2026-02-20T17:17:49.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705813 2026-02-20T17:17:49.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631 1-38654705813' 2026-02-20T17:17:49.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:49.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:49.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574994 2026-02-20T17:17:49.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574994 2026-02-20T17:17:49.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631 1-38654705813 2-55834574994' 2026-02-20T17:17:49.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:49.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836631 2026-02-20T17:17:49.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:49.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:49.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836631 2026-02-20T17:17:49.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:49.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836631 2026-02-20T17:17:49.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836631' 2026-02-20T17:17:49.944 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836631 2026-02-20T17:17:49.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:50.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836631 2026-02-20T17:17:50.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:51.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:51.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:51.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836631 2026-02-20T17:17:51.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:51.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705813 2026-02-20T17:17:51.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:51.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:51.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705813 2026-02-20T17:17:51.273 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:51.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705813 2026-02-20T17:17:51.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705813' 2026-02-20T17:17:51.276 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705813 2026-02-20T17:17:51.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:51.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705813 -lt 38654705813 2026-02-20T17:17:51.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:51.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574994 2026-02-20T17:17:51.439 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:51.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:51.440 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574994 2026-02-20T17:17:51.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:51.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574994 2026-02-20T17:17:51.443 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574994 2026-02-20T17:17:51.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574994' 2026-02-20T17:17:51.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:51.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574995 -lt 55834574994 2026-02-20T17:17:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:51.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:51.758 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:51.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:51.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:51.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:51.926 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:51.926 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:51.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:51.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:51.927 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:51.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836635 2026-02-20T17:17:51.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836635 2026-02-20T17:17:51.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635' 2026-02-20T17:17:51.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:51.998 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:52.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705816 2026-02-20T17:17:52.070 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705816 2026-02-20T17:17:52.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816' 2026-02-20T17:17:52.071 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:52.071 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:52.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574997 2026-02-20T17:17:52.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574997 2026-02-20T17:17:52.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816 2-55834574997' 2026-02-20T17:17:52.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:52.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836635 2026-02-20T17:17:52.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:52.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:52.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836635 2026-02-20T17:17:52.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:52.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836635 2026-02-20T17:17:52.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836635' 2026-02-20T17:17:52.147 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836635 2026-02-20T17:17:52.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:52.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836635 2026-02-20T17:17:52.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:53.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:53.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:53.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836635 2026-02-20T17:17:53.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:53.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705816 2026-02-20T17:17:53.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:53.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:53.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705816 2026-02-20T17:17:53.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:53.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705816 2026-02-20T17:17:53.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705816' 2026-02-20T17:17:53.479 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705816 2026-02-20T17:17:53.479 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:53.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705816 -lt 38654705816 2026-02-20T17:17:53.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574997 2026-02-20T17:17:53.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:53.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:53.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574997 2026-02-20T17:17:53.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:53.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574997 2026-02-20T17:17:53.635 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834574997 2026-02-20T17:17:53.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574997' 2026-02-20T17:17:53.636 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:53.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574998 -lt 55834574997 2026-02-20T17:17:53.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:53.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:53.805 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:53.952 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:53.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:53.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:53.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:53.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:53.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:54.116 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:54.116 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:54.117 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:54.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:54.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:54.117 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:54.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836638 2026-02-20T17:17:54.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836638 2026-02-20T17:17:54.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638' 2026-02-20T17:17:54.188 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:54.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705819 2026-02-20T17:17:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705819 2026-02-20T17:17:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-38654705819' 2026-02-20T17:17:54.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:54.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:54.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575000 2026-02-20T17:17:54.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575000 2026-02-20T17:17:54.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-38654705819 2-55834575000' 2026-02-20T17:17:54.331 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:54.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836638 2026-02-20T17:17:54.331 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:54.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:54.332 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836638 2026-02-20T17:17:54.333 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:54.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836638 2026-02-20T17:17:54.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836638' 2026-02-20T17:17:54.335 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836638 2026-02-20T17:17:54.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:54.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836638 2026-02-20T17:17:54.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:54.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705819 2026-02-20T17:17:54.494 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:54.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:54.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705819 2026-02-20T17:17:54.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:54.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705819 2026-02-20T17:17:54.498 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705819 2026-02-20T17:17:54.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705819' 2026-02-20T17:17:54.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:54.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705819 -lt 38654705819 2026-02-20T17:17:54.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:54.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575000 2026-02-20T17:17:54.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:54.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:54.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575000 2026-02-20T17:17:54.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:54.669 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575000 2026-02-20T17:17:54.669 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575000 2026-02-20T17:17:54.670 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575000' 2026-02-20T17:17:54.670 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:54.829 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575001 -lt 55834575000 2026-02-20T17:17:54.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:54.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:54.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:54.975 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:54.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:54.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:54.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:54.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:54.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:55.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:55.145 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:55.145 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:55.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:55.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:55.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:55.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836640 2026-02-20T17:17:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836640 2026-02-20T17:17:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640' 2026-02-20T17:17:55.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:55.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:55.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705821 2026-02-20T17:17:55.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705821 2026-02-20T17:17:55.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705821' 2026-02-20T17:17:55.294 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:55.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:55.367 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575003 2026-02-20T17:17:55.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575003 2026-02-20T17:17:55.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705821 2-55834575003' 2026-02-20T17:17:55.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:55.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836640 2026-02-20T17:17:55.368 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:55.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:55.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836640 2026-02-20T17:17:55.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:55.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836640 2026-02-20T17:17:55.371 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836640 2026-02-20T17:17:55.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836640' 2026-02-20T17:17:55.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:55.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836640 2026-02-20T17:17:55.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:56.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:56.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:56.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836640 2026-02-20T17:17:56.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:56.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705821 2026-02-20T17:17:56.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:56.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:56.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705821 2026-02-20T17:17:56.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:56.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705821 2026-02-20T17:17:56.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705821' 2026-02-20T17:17:56.699 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705821 2026-02-20T17:17:56.699 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:56.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705822 -lt 38654705821 2026-02-20T17:17:56.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:56.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575003 2026-02-20T17:17:56.865 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:56.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:56.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575003 2026-02-20T17:17:56.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:56.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575003 2026-02-20T17:17:56.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575003' 2026-02-20T17:17:56.868 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575003 2026-02-20T17:17:56.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:57.030 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575004 -lt 55834575003 2026-02-20T17:17:57.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:57.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:57.031 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:57.178 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:57.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:57.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:57.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:57.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:57.189 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:57.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:57.346 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:57.346 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:57.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:57.346 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:57.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:57.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836643 2026-02-20T17:17:57.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836643 2026-02-20T17:17:57.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643' 2026-02-20T17:17:57.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:57.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705824 2026-02-20T17:17:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705824 2026-02-20T17:17:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705824' 2026-02-20T17:17:57.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:57.491 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:57.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575006 2026-02-20T17:17:57.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575006 2026-02-20T17:17:57.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705824 2-55834575006' 2026-02-20T17:17:57.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:57.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836643 2026-02-20T17:17:57.564 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:57.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:57.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836643 2026-02-20T17:17:57.566 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:57.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836643 2026-02-20T17:17:57.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836643' 2026-02-20T17:17:57.567 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836643 2026-02-20T17:17:57.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:57.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836643 2026-02-20T17:17:57.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:17:58.737 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:17:58.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:58.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836643 2026-02-20T17:17:58.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:58.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705824 2026-02-20T17:17:58.895 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:58.897 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:17:58.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705824 2026-02-20T17:17:58.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:58.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705824 2026-02-20T17:17:58.899 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705824 2026-02-20T17:17:58.899 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705824' 2026-02-20T17:17:58.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:17:59.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705825 -lt 38654705824 2026-02-20T17:17:59.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:59.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575006 2026-02-20T17:17:59.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:59.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:17:59.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575006 2026-02-20T17:17:59.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:59.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575006 2026-02-20T17:17:59.069 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575006 2026-02-20T17:17:59.069 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575006' 2026-02-20T17:17:59.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:17:59.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575007 -lt 55834575006 2026-02-20T17:17:59.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:17:59.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:17:59.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:17:59.380 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:17:59.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:17:59.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:17:59.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:17:59.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:17:59.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:17:59.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:17:59.547 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:17:59.547 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:17:59.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:17:59.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:59.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:17:59.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836646 2026-02-20T17:17:59.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836646 2026-02-20T17:17:59.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646' 2026-02-20T17:17:59.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:59.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:17:59.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705827 2026-02-20T17:17:59.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705827 2026-02-20T17:17:59.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705827' 2026-02-20T17:17:59.692 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:17:59.692 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:17:59.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575009 2026-02-20T17:17:59.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575009 2026-02-20T17:17:59.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705827 2-55834575009' 2026-02-20T17:17:59.764 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:17:59.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836646 2026-02-20T17:17:59.764 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:17:59.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:17:59.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836646 2026-02-20T17:17:59.766 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:17:59.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836646 2026-02-20T17:17:59.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836646' 2026-02-20T17:17:59.768 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836646 2026-02-20T17:17:59.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:17:59.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836646 2026-02-20T17:17:59.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:00.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:00.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:01.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836646 2026-02-20T17:18:01.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:01.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705827 2026-02-20T17:18:01.097 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:01.098 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:01.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705827 2026-02-20T17:18:01.099 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:01.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705827 2026-02-20T17:18:01.101 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705827 2026-02-20T17:18:01.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705827' 2026-02-20T17:18:01.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:01.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705828 -lt 38654705827 2026-02-20T17:18:01.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:01.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575009 2026-02-20T17:18:01.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:01.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:01.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575009 2026-02-20T17:18:01.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:01.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575009 2026-02-20T17:18:01.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575009' 2026-02-20T17:18:01.262 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575009 2026-02-20T17:18:01.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:01.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575010 -lt 55834575009 2026-02-20T17:18:01.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:01.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:01.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:01.575 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:01.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:01.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:01.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:01.584 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:01.584 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:01.740 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:01.740 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:01.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:01.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836650 2026-02-20T17:18:01.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836650 2026-02-20T17:18:01.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650' 2026-02-20T17:18:01.814 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:01.814 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:01.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705831 2026-02-20T17:18:01.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705831 2026-02-20T17:18:01.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-38654705831' 2026-02-20T17:18:01.886 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:01.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:01.960 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575012 2026-02-20T17:18:01.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575012 2026-02-20T17:18:01.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-38654705831 2-55834575012' 2026-02-20T17:18:01.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:01.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836650 2026-02-20T17:18:01.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:01.962 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:01.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836650 2026-02-20T17:18:01.963 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:01.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836650 2026-02-20T17:18:01.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836650' 2026-02-20T17:18:01.965 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836650 2026-02-20T17:18:01.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:02.127 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836650 2026-02-20T17:18:02.128 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:03.129 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:03.129 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:03.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836650 2026-02-20T17:18:03.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:03.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705831 2026-02-20T17:18:03.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:03.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:03.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705831 2026-02-20T17:18:03.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:03.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705831 2026-02-20T17:18:03.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705831' 2026-02-20T17:18:03.292 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705831 2026-02-20T17:18:03.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:03.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705831 -lt 38654705831 2026-02-20T17:18:03.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:03.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575012 2026-02-20T17:18:03.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:03.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:03.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575012 2026-02-20T17:18:03.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:03.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575012 2026-02-20T17:18:03.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575012' 2026-02-20T17:18:03.461 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575012 2026-02-20T17:18:03.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:03.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575013 -lt 55834575012 2026-02-20T17:18:03.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:03.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:03.622 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:03.772 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:03.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:03.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:03.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:03.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:03.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:03.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:03.939 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:03.939 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:03.940 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:04.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836653 2026-02-20T17:18:04.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836653 2026-02-20T17:18:04.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653' 2026-02-20T17:18:04.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:04.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:04.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705834 2026-02-20T17:18:04.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705834 2026-02-20T17:18:04.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834' 2026-02-20T17:18:04.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:04.083 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:04.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575015 2026-02-20T17:18:04.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575015 2026-02-20T17:18:04.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834 2-55834575015' 2026-02-20T17:18:04.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:04.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836653 2026-02-20T17:18:04.154 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:04.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:04.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836653 2026-02-20T17:18:04.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:04.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836653 2026-02-20T17:18:04.158 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836653' 2026-02-20T17:18:04.159 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836653 2026-02-20T17:18:04.159 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:04.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836653 2026-02-20T17:18:04.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:05.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:05.324 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:05.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836653 2026-02-20T17:18:05.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:05.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705834 2026-02-20T17:18:05.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:05.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:05.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705834 2026-02-20T17:18:05.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:05.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705834 2026-02-20T17:18:05.486 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705834 2026-02-20T17:18:05.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705834' 2026-02-20T17:18:05.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:05.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705834 -lt 38654705834 2026-02-20T17:18:05.649 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:05.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575015 2026-02-20T17:18:05.650 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:05.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:05.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575015 2026-02-20T17:18:05.652 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:05.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575015 2026-02-20T17:18:05.653 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575015 2026-02-20T17:18:05.654 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575015' 2026-02-20T17:18:05.654 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:05.817 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575016 -lt 55834575015 2026-02-20T17:18:05.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:05.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:05.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:05.966 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:05.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:05.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:05.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:05.976 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:05.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:06.130 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836656 2026-02-20T17:18:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836656 2026-02-20T17:18:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656' 2026-02-20T17:18:06.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:06.203 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:06.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705837 2026-02-20T17:18:06.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705837 2026-02-20T17:18:06.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-38654705837' 2026-02-20T17:18:06.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:06.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575019 2026-02-20T17:18:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575019 2026-02-20T17:18:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-38654705837 2-55834575019' 2026-02-20T17:18:06.349 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:06.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836656 2026-02-20T17:18:06.350 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:06.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:06.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836656 2026-02-20T17:18:06.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:06.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836656 2026-02-20T17:18:06.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836656' 2026-02-20T17:18:06.353 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836656 2026-02-20T17:18:06.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:06.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836656 2026-02-20T17:18:06.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:06.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705837 2026-02-20T17:18:06.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:06.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:06.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705837 2026-02-20T17:18:06.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:06.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705837 2026-02-20T17:18:06.513 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705837 2026-02-20T17:18:06.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705837' 2026-02-20T17:18:06.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:06.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705837 -lt 38654705837 2026-02-20T17:18:06.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:06.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575019 2026-02-20T17:18:06.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:06.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:06.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575019 2026-02-20T17:18:06.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:06.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575019 2026-02-20T17:18:06.686 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575019' 2026-02-20T17:18:06.687 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575019 2026-02-20T17:18:06.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:06.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575019 -lt 55834575019 2026-02-20T17:18:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:06.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:07.001 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:07.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:07.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:07.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:07.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:07.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:07.172 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-20T17:18:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-20T17:18:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-20T17:18:07.245 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:07.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:07.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705839 2026-02-20T17:18:07.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705839 2026-02-20T17:18:07.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839' 2026-02-20T17:18:07.315 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:07.315 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575021 2026-02-20T17:18:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575021 2026-02-20T17:18:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839 2-55834575021' 2026-02-20T17:18:07.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:07.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-20T17:18:07.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:07.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:07.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-20T17:18:07.389 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:07.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-20T17:18:07.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-20T17:18:07.391 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836658 2026-02-20T17:18:07.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:07.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836658 2026-02-20T17:18:07.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:08.557 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:08.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:08.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836658 2026-02-20T17:18:08.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:08.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705839 2026-02-20T17:18:08.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:08.719 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:08.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705839 2026-02-20T17:18:08.720 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:08.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705839 2026-02-20T17:18:08.721 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705839' 2026-02-20T17:18:08.722 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705839 2026-02-20T17:18:08.722 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:08.884 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705840 -lt 38654705839 2026-02-20T17:18:08.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:08.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575021 2026-02-20T17:18:08.885 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:08.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:08.887 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575021 2026-02-20T17:18:08.888 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:08.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575021 2026-02-20T17:18:08.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575021' 2026-02-20T17:18:08.889 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575021 2026-02-20T17:18:08.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:09.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575022 -lt 55834575021 2026-02-20T17:18:09.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:09.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:09.050 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:09.194 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:09.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:09.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:09.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:09.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:09.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:09.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:09.355 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:09.355 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:09.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:09.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:09.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:09.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-20T17:18:09.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-20T17:18:09.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-20T17:18:09.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:09.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705842 2026-02-20T17:18:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705842 2026-02-20T17:18:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842' 2026-02-20T17:18:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:09.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:09.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575024 2026-02-20T17:18:09.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575024 2026-02-20T17:18:09.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842 2-55834575024' 2026-02-20T17:18:09.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:09.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-20T17:18:09.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:09.572 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:09.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-20T17:18:09.573 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:09.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-20T17:18:09.574 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836661 2026-02-20T17:18:09.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-20T17:18:09.575 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:09.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836661 2026-02-20T17:18:09.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:10.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:10.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:10.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836661 2026-02-20T17:18:10.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:10.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705842 2026-02-20T17:18:10.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:10.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:10.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705842 2026-02-20T17:18:10.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705842 2026-02-20T17:18:10.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705842' 2026-02-20T17:18:10.906 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705842 2026-02-20T17:18:10.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:11.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705843 -lt 38654705842 2026-02-20T17:18:11.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:11.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575024 2026-02-20T17:18:11.060 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:11.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:11.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575024 2026-02-20T17:18:11.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:11.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575024 2026-02-20T17:18:11.064 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575024 2026-02-20T17:18:11.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575024' 2026-02-20T17:18:11.064 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:11.230 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575025 -lt 55834575024 2026-02-20T17:18:11.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:11.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:11.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:11.380 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:11.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:11.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:11.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:11.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:11.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:11.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:11.546 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:11.546 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:11.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:11.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:11.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:11.620 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-20T17:18:11.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-20T17:18:11.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-20T17:18:11.621 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:11.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:11.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705845 2026-02-20T17:18:11.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705845 2026-02-20T17:18:11.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845' 2026-02-20T17:18:11.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:11.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:11.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575027 2026-02-20T17:18:11.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575027 2026-02-20T17:18:11.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845 2-55834575027' 2026-02-20T17:18:11.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:11.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-20T17:18:11.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:11.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:11.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-20T17:18:11.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:11.770 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-20T17:18:11.770 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836664 2026-02-20T17:18:11.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-20T17:18:11.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:11.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836664 2026-02-20T17:18:11.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:12.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:12.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:13.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836664 2026-02-20T17:18:13.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:13.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705845 2026-02-20T17:18:13.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:13.101 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:13.101 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705845 2026-02-20T17:18:13.102 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705845 2026-02-20T17:18:13.103 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705845' 2026-02-20T17:18:13.104 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705845 2026-02-20T17:18:13.104 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:13.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705846 -lt 38654705845 2026-02-20T17:18:13.273 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:13.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575027 2026-02-20T17:18:13.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:13.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:13.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575027 2026-02-20T17:18:13.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:13.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575027 2026-02-20T17:18:13.278 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575027' 2026-02-20T17:18:13.278 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575027 2026-02-20T17:18:13.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:13.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575028 -lt 55834575027 2026-02-20T17:18:13.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:13.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:13.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:13.597 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:13.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:13.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:13.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:13.608 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:13.608 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:13.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:13.767 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:13.767 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:13.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:13.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:13.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836668 2026-02-20T17:18:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836668 2026-02-20T17:18:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668' 2026-02-20T17:18:13.842 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:13.842 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:13.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705849 2026-02-20T17:18:13.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705849 2026-02-20T17:18:13.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849' 2026-02-20T17:18:13.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:13.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:13.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575030 2026-02-20T17:18:13.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575030 2026-02-20T17:18:13.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849 2-55834575030' 2026-02-20T17:18:13.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:13.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836668 2026-02-20T17:18:13.988 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:13.989 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:13.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836668 2026-02-20T17:18:13.990 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:13.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836668 2026-02-20T17:18:13.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836668' 2026-02-20T17:18:13.992 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836668 2026-02-20T17:18:13.992 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:14.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836668 2026-02-20T17:18:14.156 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:15.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:15.157 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:15.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836668 2026-02-20T17:18:15.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:15.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705849 2026-02-20T17:18:15.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:15.319 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:15.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705849 2026-02-20T17:18:15.320 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:15.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705849 2026-02-20T17:18:15.321 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705849 2026-02-20T17:18:15.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705849' 2026-02-20T17:18:15.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705849 -lt 38654705849 2026-02-20T17:18:15.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:15.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575030 2026-02-20T17:18:15.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:15.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:15.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575030 2026-02-20T17:18:15.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:15.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575030 2026-02-20T17:18:15.491 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575030 2026-02-20T17:18:15.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575030' 2026-02-20T17:18:15.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:15.652 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575031 -lt 55834575030 2026-02-20T17:18:15.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:15.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:15.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:15.799 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:15.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:15.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:15.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:15.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:15.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:15.971 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:15.972 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:15.972 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:15.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:15.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:15.972 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:16.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836671 2026-02-20T17:18:16.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836671 2026-02-20T17:18:16.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671' 2026-02-20T17:18:16.043 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:16.043 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:16.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705852 2026-02-20T17:18:16.114 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705852 2026-02-20T17:18:16.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-38654705852' 2026-02-20T17:18:16.115 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:16.115 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575033 2026-02-20T17:18:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575033 2026-02-20T17:18:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-38654705852 2-55834575033' 2026-02-20T17:18:16.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:16.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836671 2026-02-20T17:18:16.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:16.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:16.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836671 2026-02-20T17:18:16.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:16.189 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836671 2026-02-20T17:18:16.189 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836671 2026-02-20T17:18:16.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836671' 2026-02-20T17:18:16.190 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:16.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836671 2026-02-20T17:18:16.351 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:17.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:17.352 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:17.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836671 -lt 21474836671 2026-02-20T17:18:17.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:17.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705852 2026-02-20T17:18:17.510 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:17.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:17.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705852 2026-02-20T17:18:17.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:17.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705852 2026-02-20T17:18:17.513 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705852' 2026-02-20T17:18:17.514 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705852 2026-02-20T17:18:17.514 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:17.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705852 -lt 38654705852 2026-02-20T17:18:17.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:17.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575033 2026-02-20T17:18:17.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:17.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:17.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575033 2026-02-20T17:18:17.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:17.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575033 2026-02-20T17:18:17.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575033' 2026-02-20T17:18:17.687 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575033 2026-02-20T17:18:17.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:17.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575034 -lt 55834575033 2026-02-20T17:18:17.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:17.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:17.852 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:17.997 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:18.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:18.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:18.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:18.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:18.009 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:18.173 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:18.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836674 2026-02-20T17:18:18.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836674 2026-02-20T17:18:18.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674' 2026-02-20T17:18:18.247 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:18.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:18.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705855 2026-02-20T17:18:18.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705855 2026-02-20T17:18:18.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-38654705855' 2026-02-20T17:18:18.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:18.321 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:18.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575037 2026-02-20T17:18:18.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575037 2026-02-20T17:18:18.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-38654705855 2-55834575037' 2026-02-20T17:18:18.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:18.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836674 2026-02-20T17:18:18.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:18.392 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:18.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836674 2026-02-20T17:18:18.393 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:18.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836674 2026-02-20T17:18:18.394 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836674 2026-02-20T17:18:18.395 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836674' 2026-02-20T17:18:18.395 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:18.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836674 -lt 21474836674 2026-02-20T17:18:18.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:18.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705855 2026-02-20T17:18:18.563 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:18.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:18.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705855 2026-02-20T17:18:18.565 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:18.566 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705855 2026-02-20T17:18:18.567 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705855 2026-02-20T17:18:18.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705855' 2026-02-20T17:18:18.567 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:18.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705855 -lt 38654705855 2026-02-20T17:18:18.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:18.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575037 2026-02-20T17:18:18.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:18.733 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:18.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575037 2026-02-20T17:18:18.734 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:18.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575037 2026-02-20T17:18:18.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575037' 2026-02-20T17:18:18.736 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575037 2026-02-20T17:18:18.736 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:18.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575036 -lt 55834575037 2026-02-20T17:18:18.907 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:19.908 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:19.908 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:20.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575036 -lt 55834575037 2026-02-20T17:18:20.064 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:21.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:18:21.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:21.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575039 -lt 55834575037 2026-02-20T17:18:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:21.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:21.372 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:21.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:21.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:21.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:21.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:21.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:21.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:21.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836678 2026-02-20T17:18:21.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836678 2026-02-20T17:18:21.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678' 2026-02-20T17:18:21.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:21.611 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:21.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705859 2026-02-20T17:18:21.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705859 2026-02-20T17:18:21.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859' 2026-02-20T17:18:21.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:21.683 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:21.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575041 2026-02-20T17:18:21.753 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575041 2026-02-20T17:18:21.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859 2-55834575041' 2026-02-20T17:18:21.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:21.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836678 2026-02-20T17:18:21.754 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:21.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:21.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836678 2026-02-20T17:18:21.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:21.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836678 2026-02-20T17:18:21.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836678' 2026-02-20T17:18:21.758 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836678 2026-02-20T17:18:21.758 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:21.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836676 -lt 21474836678 2026-02-20T17:18:21.925 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:22.926 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:22.926 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:23.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836678 2026-02-20T17:18:23.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:23.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705859 2026-02-20T17:18:23.087 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:23.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:23.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705859 2026-02-20T17:18:23.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:23.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705859 2026-02-20T17:18:23.091 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705859' 2026-02-20T17:18:23.091 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705859 2026-02-20T17:18:23.092 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:23.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705860 -lt 38654705859 2026-02-20T17:18:23.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:23.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575041 2026-02-20T17:18:23.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:23.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:23.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575041 2026-02-20T17:18:23.260 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:23.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575041 2026-02-20T17:18:23.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575041' 2026-02-20T17:18:23.262 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575041 2026-02-20T17:18:23.262 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:23.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575042 -lt 55834575041 2026-02-20T17:18:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:23.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:23.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:23.576 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:23.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:23.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:23.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:23.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:23.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:23.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:23.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836681 2026-02-20T17:18:23.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836681 2026-02-20T17:18:23.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681' 2026-02-20T17:18:23.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:23.818 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:23.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705863 2026-02-20T17:18:23.890 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705863 2026-02-20T17:18:23.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681 1-38654705863' 2026-02-20T17:18:23.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:23.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:23.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575044 2026-02-20T17:18:23.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575044 2026-02-20T17:18:23.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681 1-38654705863 2-55834575044' 2026-02-20T17:18:23.961 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:23.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836681 2026-02-20T17:18:23.962 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:23.963 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:23.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836681 2026-02-20T17:18:23.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:23.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836681 2026-02-20T17:18:23.966 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836681 2026-02-20T17:18:23.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836681' 2026-02-20T17:18:23.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:24.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836681 2026-02-20T17:18:24.132 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:25.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:25.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:25.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836681 2026-02-20T17:18:25.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:25.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705863 2026-02-20T17:18:25.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:25.289 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:25.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705863 2026-02-20T17:18:25.290 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:25.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705863 2026-02-20T17:18:25.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705863' 2026-02-20T17:18:25.292 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705863 2026-02-20T17:18:25.292 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:25.460 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705863 -lt 38654705863 2026-02-20T17:18:25.461 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:25.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575044 2026-02-20T17:18:25.461 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:25.462 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:25.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575044 2026-02-20T17:18:25.463 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:25.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575044 2026-02-20T17:18:25.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575044' 2026-02-20T17:18:25.465 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575044 2026-02-20T17:18:25.465 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:25.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575045 -lt 55834575044 2026-02-20T17:18:25.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:25.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:25.624 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:25.768 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:25.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:25.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:25.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:25.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:25.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:25.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:25.931 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:25.932 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:25.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:25.932 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:25.932 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:26.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836685 2026-02-20T17:18:26.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836685 2026-02-20T17:18:26.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685' 2026-02-20T17:18:26.004 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:26.004 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:26.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705866 2026-02-20T17:18:26.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705866 2026-02-20T17:18:26.073 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705866' 2026-02-20T17:18:26.074 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:26.074 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:26.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575047 2026-02-20T17:18:26.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575047 2026-02-20T17:18:26.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705866 2-55834575047' 2026-02-20T17:18:26.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:26.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836685 2026-02-20T17:18:26.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:26.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:26.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836685 2026-02-20T17:18:26.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836685 2026-02-20T17:18:26.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836685' 2026-02-20T17:18:26.151 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836685 2026-02-20T17:18:26.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:26.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836685 2026-02-20T17:18:26.305 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:27.306 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:27.307 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:27.468 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836685 -lt 21474836685 2026-02-20T17:18:27.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:27.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705866 2026-02-20T17:18:27.469 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:27.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:27.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705866 2026-02-20T17:18:27.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:27.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705866 2026-02-20T17:18:27.473 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705866 2026-02-20T17:18:27.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705866' 2026-02-20T17:18:27.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:27.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705866 -lt 38654705866 2026-02-20T17:18:27.629 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:27.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575047 2026-02-20T17:18:27.630 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:27.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:27.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575047 2026-02-20T17:18:27.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:27.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575047 2026-02-20T17:18:27.633 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575047 2026-02-20T17:18:27.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575047' 2026-02-20T17:18:27.634 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:27.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575048 -lt 55834575047 2026-02-20T17:18:27.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:27.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:27.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:27.935 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:27.945 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:27.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:28.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:28.108 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:28.108 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:28.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:28.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:28.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:28.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836688 2026-02-20T17:18:28.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836688 2026-02-20T17:18:28.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688' 2026-02-20T17:18:28.179 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:28.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705869 2026-02-20T17:18:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705869 2026-02-20T17:18:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-38654705869' 2026-02-20T17:18:28.251 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:28.251 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:28.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575050 2026-02-20T17:18:28.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575050 2026-02-20T17:18:28.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-38654705869 2-55834575050' 2026-02-20T17:18:28.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:28.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836688 2026-02-20T17:18:28.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:28.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:28.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836688 2026-02-20T17:18:28.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:28.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836688 2026-02-20T17:18:28.329 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836688 2026-02-20T17:18:28.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836688' 2026-02-20T17:18:28.329 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:28.481 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836688 2026-02-20T17:18:28.482 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:28.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705869 2026-02-20T17:18:28.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:28.483 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:28.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705869 2026-02-20T17:18:28.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:28.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705869 2026-02-20T17:18:28.486 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705869 2026-02-20T17:18:28.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705869' 2026-02-20T17:18:28.486 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:28.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705869 -lt 38654705869 2026-02-20T17:18:28.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:28.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575050 2026-02-20T17:18:28.642 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:28.644 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:28.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575050 2026-02-20T17:18:28.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:28.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575050 2026-02-20T17:18:28.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575050' 2026-02-20T17:18:28.646 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575050 2026-02-20T17:18:28.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:28.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575051 -lt 55834575050 2026-02-20T17:18:28.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:28.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:28.804 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:28.954 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:28.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-20T17:18:28.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-20T17:18:28.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-20T17:18:28.964 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:18:28.964 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:29.122 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:18:29.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836690 2026-02-20T17:18:29.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836690 2026-02-20T17:18:29.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690' 2026-02-20T17:18:29.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:29.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:18:29.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705871 2026-02-20T17:18:29.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705871 2026-02-20T17:18:29.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705871' 2026-02-20T17:18:29.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:18:29.267 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:18:29.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575052 2026-02-20T17:18:29.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575052 2026-02-20T17:18:29.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705871 2-55834575052' 2026-02-20T17:18:29.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:29.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836690 2026-02-20T17:18:29.342 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:29.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:18:29.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836690 2026-02-20T17:18:29.343 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:29.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836690 2026-02-20T17:18:29.345 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836690' 2026-02-20T17:18:29.346 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836690 2026-02-20T17:18:29.346 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:29.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836690 2026-02-20T17:18:29.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:18:30.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:18:30.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:18:30.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836690 2026-02-20T17:18:30.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:30.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705871 2026-02-20T17:18:30.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:30.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:18:30.663 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705871 2026-02-20T17:18:30.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:30.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705871 2026-02-20T17:18:30.665 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705871' 2026-02-20T17:18:30.665 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705871 2026-02-20T17:18:30.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:18:30.833 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705872 -lt 38654705871 2026-02-20T17:18:30.834 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:18:30.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575052 2026-02-20T17:18:30.834 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:18:30.835 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:18:30.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575052 2026-02-20T17:18:30.836 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:18:30.837 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575052 2026-02-20T17:18:30.838 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575052' 2026-02-20T17:18:30.838 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 55834575052 2026-02-20T17:18:30.838 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:18:30.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575054 -lt 55834575052 2026-02-20T17:18:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-20T17:18:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-20T17:18:30.997 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-20T17:18:31.141 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:18:31.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-20T17:18:31.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-20T17:18:31.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-20T17:18:36.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'noscrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-20T17:18:36.161 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:16:34.565+0000 7f472da56640 10 osd.1 pg_epoch: 17 pg[1.0( v 16'1000 (0'0,16'1000] local-lis/les=14/15 n=1000 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0,2] r=0 lpr=14 crt=16'1000 lcod 16'999 mlcod 16'999 active+clean+scrubbing(periodic-regular)] scrubber: noscrub set, aborting 2026-02-20T17:18:36.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-20T17:18:36.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:36.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:36.162 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:36.163 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:36.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-20T17:18:36.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset noscrub 2026-02-20T17:18:36.677 INFO:tasks.workunit.client.0.trial013.stderr:noscrub is unset 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:36.688 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:36.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:36.689 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:36.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:36.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:37.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:37.853 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:37.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:37.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:37.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:37.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:37.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:38.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:38.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:39.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:39.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:39.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:39.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:39.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:39.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:39.016 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:39.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:39.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:40.180 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:40.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:40.337 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:40.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:41.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:41.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:41.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:41.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:41.339 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:41.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:41.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:41.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:41.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:42.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:42.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:42.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:42.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:42.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:42.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:42.499 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:42.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:42.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:43.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:43.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:43.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:43.655 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:43.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:43.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:43.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:44.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:44.820 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:44.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:44.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:44.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:44.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:44.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:44.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:44.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:45.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:45.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:45.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:45.983 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:45.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:45.984 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:46.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:46.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:47.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:47.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:47.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:47.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:47.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:47.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:47.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:47.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:48.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:48.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:48.297 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:48.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:48.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:49.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:49.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:49.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:49.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:49.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:49.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:49.453 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:49.609 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:49.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:50.610 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:50.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:50.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:50.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:50.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:50.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:50.612 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:50.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:50.765 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:51.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:51.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:51.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:51.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:51.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:51.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:51.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:51.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:51.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:52.922 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:52.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:52.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:52.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:52.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:52.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:52.924 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:53.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:53.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:54.080 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:54.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:54.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:54.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:54.081 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:54.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:54.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:54.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:54.234 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:55.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:55.235 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:55.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:55.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:55.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:55.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:55.236 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:55.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:55.388 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:56.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:56.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:56.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:56.390 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:56.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:56.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:56.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:56.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:56.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:57.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:57.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:57.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:57.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:57.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:57.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:57.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:57.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:57.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:58.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:58.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:58.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:58.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:58.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:58.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:58.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:18:58.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:18:58.871 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:18:59.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:18:59.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:18:59.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:18:59.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:18:59.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:18:59.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:18:59.873 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:00.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:00.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:01.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:01.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:01.034 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:01.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:01.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:01.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:01.035 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:01.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:01.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:02.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:02.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:02.194 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:02.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:02.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:02.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:02.195 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:02.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:02.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:03.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:03.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:03.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:03.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:03.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:03.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:03.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:03.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:03.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:04.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:04.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:04.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:04.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:04.508 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:04.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:04.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:04.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:04.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:05.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:05.664 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:05.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:05.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:05.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:05.665 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:05.666 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:05.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:05.818 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:06.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:06.819 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:06.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:06.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:06.820 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:06.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:06.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:06.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:06.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:07.974 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:07.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:07.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:07.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:07.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:07.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:07.977 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:08.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:08.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:09.133 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:09.134 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:09.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:09.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:09.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:09.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:09.135 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:09.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:09.291 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:10.292 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:10.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:10.293 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:10.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:10.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:10.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:10.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:10.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:10.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:11.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:11.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:11.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:11.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:11.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:11.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:11.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:11.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:11.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:12.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:12.601 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:12.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:12.602 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:12.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:12.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:12.603 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:12.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:13.757 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:13.758 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:13.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:13.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:13.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:13.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:13.759 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:13.910 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:13.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:14.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:14.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:14.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:14.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:14.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:14.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:14.913 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:15.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:15.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:16.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:16.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:16.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:16.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:16.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:16.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:16.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:16.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:16.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:17.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:17.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:17.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:17.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:17.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:17.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:17.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:17.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:17.389 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:18.390 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:18.391 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:18.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:18.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:18.391 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:18.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:18.392 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:18.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:18.549 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:19.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:19.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:19.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:19.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:19.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:19.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:19.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:19.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:19.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:20.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:20.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:20.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:20.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:20.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:20.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:20.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:20.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:20.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:21.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:21.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:21.867 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:21.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:21.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:21.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:21.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:22.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:22.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:23.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:23.023 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:23.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:23.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:23.177 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:24.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:24.178 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:24.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:24.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:24.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:24.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:24.179 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:24.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:25.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:25.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:25.334 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:25.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:25.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:25.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:25.335 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:25.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:25.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:26.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:26.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:26.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:26.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:26.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:26.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:26.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:26.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:26.647 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:27.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:27.648 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:27.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:27.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:27.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:27.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:27.649 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:27.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:27.800 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:28.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:28.801 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:28.802 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:28.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:28.955 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:29.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:29.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:29.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:29.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:29.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:29.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:29.958 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:30.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:30.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:31.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:31.110 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:31.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:31.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:31.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:31.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:31.111 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:31.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:31.266 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:32.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:32.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:32.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:32.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:32.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:32.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:32.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:32.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:33.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:33.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:33.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:33.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:33.422 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:33.423 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:33.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:33.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:34.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:34.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:34.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:34.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:34.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:34.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:34.583 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:34.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:34.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:35.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:35.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:35.742 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:35.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:35.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:36.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:36.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:36.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:36.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:36.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:36.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:36.903 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:37.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:37.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:38.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:38.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:38.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:38.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:38.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:38.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:38.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:38.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:38.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:39.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:39.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:39.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:39.224 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:39.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:39.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:40.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:40.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:40.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:40.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:40.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:40.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:40.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:40.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:40.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:41.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:41.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:41.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:41.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:41.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:41.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:41.545 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:41.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:41.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:42.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:42.703 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:42.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:42.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:42.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:42.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:42.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:42.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:42.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:43.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:43.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:43.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:43.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:43.863 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:43.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:43.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:44.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:44.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:45.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:45.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:45.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:45.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:45.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:45.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:45.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:45.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:45.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:46.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:46.181 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:46.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:46.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:46.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:46.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:46.183 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:46.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:46.338 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:47.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:47.339 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:47.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:47.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:47.340 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:47.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:47.341 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:47.503 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:47.504 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:48.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:48.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:48.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:48.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:48.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:48.506 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:48.507 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:48.666 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:49.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:49.667 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:49.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:49.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:49.667 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:49.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:49.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:49.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:49.826 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:50.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:50.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:50.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:50.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:50.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:50.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:50.829 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:50.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:50.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:51.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:51.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:51.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:52.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:52.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:53.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:53.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:53.142 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:53.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:53.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:53.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:53.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:53.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:53.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:54.299 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:54.300 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:54.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:54.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:54.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:54.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:54.301 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:54.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:54.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:55.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:55.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:55.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:55.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:55.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:55.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:55.458 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:55.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:55.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:56.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:56.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:56.616 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:56.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:56.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:56.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:56.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:56.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:57.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:57.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:57.776 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:57.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:57.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:57.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:57.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:57.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:57.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:19:58.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:19:58.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:19:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:19:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:19:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:19:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:19:58.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:19:59.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:19:59.096 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:00.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:00.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:00.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:01.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:01.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:01.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:01.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:01.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:01.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:01.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:01.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:01.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:02.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:02.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:02.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:02.420 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:02.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:02.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:02.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:02.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:02.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:03.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:03.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:03.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:03.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:03.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:03.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:03.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:03.738 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:03.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:04.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:04.739 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:04.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:04.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:04.740 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:04.741 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:04.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:04.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:05.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:05.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:05.904 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:05.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:05.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:05.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:05.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:06.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:06.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:07.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:07.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:07.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:07.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:07.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:07.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:07.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:07.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:07.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:08.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:08.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:08.223 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:08.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-18T17:16:29.176092+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:08.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:20:09.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:20:09.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:20:09.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:20:09.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:09.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:09.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:09.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:09.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:20:07.662504+0000 '>' 2026-02-18T17:16:29.176092+0000 2026-02-20T17:20:09.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:20:09.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-20T17:20:09.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:20:09.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:20:09.543 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:20:09.544 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:20:09.544 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:09.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:20:09.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:20:09.617 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.618 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.619 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.620 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.621 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.622 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.623 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:20:09.623 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:20:09.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:09.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:20:09.623 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:20:09.690 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:20:09.690 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:20:09.690 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.690 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.691 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.692 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.693 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.694 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 11, 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:20:09.695 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:20:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:20:09.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:20:09.765 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:20:09.765 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:20:09.765 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.766 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.767 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.768 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:20:09.769 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:20:09.770 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:20:09.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_abort ------------------' 2026-02-20T17:20:09.771 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_scrub_abort ------------------ 2026-02-20T17:20:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:20:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:20:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:20:09.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:20:09.772 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:20:09.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:20:09.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:20:09.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:20:09.773 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:20:10.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:20:10.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:20:10.078 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:20:10.079 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:20:10.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:20:10.081 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:20:10.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:20:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:20:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:20:10.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:20:10.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:20:10.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:20:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:20:10.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:20:10.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:20:10.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:20:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:20:10.139 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:20:10.140 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_scrub_abort ------------------ 2026-02-20T17:20:10.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_abort ------------------' 2026-02-20T17:20:10.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:20:10.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_permit_time -------------------' 2026-02-20T17:20:10.141 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_scrub_permit_time ------------------- 2026-02-20T17:20:10.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:20:10.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:20:10.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:20:10.142 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:20:10.143 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:20:10.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:20:10.144 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:20:10.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:20:10.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:20:10.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:20:10.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:20:10.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:20:10.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:20:10.148 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:20:10.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:20:10.149 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:20:10.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:20:10.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:20:10.152 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:20:10.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:20:10.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.153 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:20:10.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:20:10.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:20:10.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:20:10.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:20:10.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.156 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.157 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:20:10.158 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:20:10.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:20:10.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:20:10.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:20:10.159 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_scrub_permit_time ----------------------- 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_permit_time -----------------------' 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_permit_time td/osd-scrub-test 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: local dir=td/osd-scrub-test 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_scrub_permit_time: local poolname=test 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_scrub_permit_time: local OSDS=3 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_scrub_permit_time: local objects=15 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:419: TEST_scrub_permit_time: TESTDATA=testdata.515468 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:421: TEST_scrub_permit_time: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:20:10.161 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:20:10.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:20:10.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:20:10.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:20:10.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.181 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.182 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:10.182 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:20:10.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:20:10.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:20:10.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:20:10.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:20:10.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:20:10.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:20:10.215 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:20:10.216 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:20:10.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:20:10.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:20:10.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:20:10.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:20:10.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.273 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:20:10.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:20:10.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:20:10.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:422: TEST_scrub_permit_time: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:20:10.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:20:10.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:20:10.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:20:10.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:20:10.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:20:10.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.425 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:10.426 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:20:10.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:20:10.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: date -d '2 hour ago' +%H 2026-02-20T17:20:10.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: sed 's/^0//' 2026-02-20T17:20:10.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: local scrub_begin_hour=15 2026-02-20T17:20:10.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: date -d '1 hour ago' +%H 2026-02-20T17:20:10.446 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: sed 's/^0//' 2026-02-20T17:20:10.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: local scrub_end_hour=16 2026-02-20T17:20:10.448 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: expr 3 - 1 2026-02-20T17:20:10.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: seq 0 2 2026-02-20T17:20:10.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:10.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 0 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:10.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:20:10.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:20:10.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:20:10.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:20:10.455 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16' 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:20:10.456 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:20:10.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c0b24ce5-1f7b-423e-a9b4-4c3662303dd8 2026-02-20T17:20:10.457 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c0b24ce5-1f7b-423e-a9b4-4c3662303dd8' 2026-02-20T17:20:10.457 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 c0b24ce5-1f7b-423e-a9b4-4c3662303dd8 2026-02-20T17:20:10.457 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:20:10.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKl5hpvv2nGxAA5Zvl23kCdC4ysCMMt5OOYg== 2026-02-20T17:20:10.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKl5hpvv2nGxAA5Zvl23kCdC4ysCMMt5OOYg=="}' 2026-02-20T17:20:10.465 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c0b24ce5-1f7b-423e-a9b4-4c3662303dd8 -i td/osd-scrub-test/0/new.json 2026-02-20T17:20:10.552 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:20:10.560 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:20:10.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 --mkfs --key AQDKl5hpvv2nGxAA5Zvl23kCdC4ysCMMt5OOYg== --osd-uuid c0b24ce5-1f7b-423e-a9b4-4c3662303dd8 2026-02-20T17:20:10.577 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:10.573+0000 7f68b35dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:10.579 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:10.573+0000 7f68b35dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:10.580 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:10.577+0000 7f68b35dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:10.580 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:10.577+0000 7f68b35dca40 -1 bdev(0x555f507e2c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:20:10.580 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:10.577+0000 7f68b35dca40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:20:11.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:20:11.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:20:11.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:20:11.342 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:20:11.342 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:20:11.447 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:20:11.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:20:11.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:11.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:20:11.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:20:11.449 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:20:11.474 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:11.469+0000 7ff8532d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:11.477 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:11.473+0000 7ff8532d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:11.478 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:11.473+0000 7ff8532d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:11.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:20:11.542 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:20:11.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:11.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:20:11.634 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:11.950 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:11.945+0000 7ff8532d8a40 -1 Falling back to public interface 2026-02-20T17:20:12.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:12.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:12.636 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:20:12.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:20:12.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:12.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:20:12.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:12.889 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:12.885+0000 7ff8532d8a40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:20:13.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:13.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:13.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:20:13.728 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:20:13.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:13.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:20:13.887 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1708609936,v1:127.0.0.1:6801/1708609936] [v2:127.0.0.1:6802/1708609936,v1:127.0.0.1:6803/1708609936] exists,up c0b24ce5-1f7b-423e-a9b4-4c3662303dd8 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 1 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:20:13.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:20:13.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:20:13.893 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16' 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:20:13.894 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:20:13.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc2959ac-5628-44c8-bbbc-8ea2a06aa177 2026-02-20T17:20:13.895 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dc2959ac-5628-44c8-bbbc-8ea2a06aa177' 2026-02-20T17:20:13.895 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 dc2959ac-5628-44c8-bbbc-8ea2a06aa177 2026-02-20T17:20:13.896 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:20:13.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNl5hpExu7NRAAdFlcxZiIpLA6KV0bBAE6kw== 2026-02-20T17:20:13.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNl5hpExu7NRAAdFlcxZiIpLA6KV0bBAE6kw=="}' 2026-02-20T17:20:13.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc2959ac-5628-44c8-bbbc-8ea2a06aa177 -i td/osd-scrub-test/1/new.json 2026-02-20T17:20:14.065 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:20:14.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:20:14.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 --mkfs --key AQDNl5hpExu7NRAAdFlcxZiIpLA6KV0bBAE6kw== --osd-uuid dc2959ac-5628-44c8-bbbc-8ea2a06aa177 2026-02-20T17:20:14.093 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.089+0000 7f5e402e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:14.094 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.089+0000 7f5e402e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:14.095 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.089+0000 7f5e402e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:14.096 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.093+0000 7f5e402e5a40 -1 bdev(0x557357650c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:20:14.096 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.093+0000 7f5e402e5a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:20:14.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:20:14.734 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:20:14.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:20:14.735 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:20:14.736 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:20:14.935 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:20:14.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:20:14.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:14.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:20:14.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:20:14.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:20:14.960 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.957+0000 7fcf89956a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:14.962 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.957+0000 7fcf89956a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:14.963 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:14.957+0000 7fcf89956a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:20:15.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:15.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:20:15.094 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:20:15.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:15.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:20:15.252 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:15.378 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:15.373+0000 7fcf89956a40 -1 Falling back to public interface 2026-02-20T17:20:16.054 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:16.049+0000 7fcf89956a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:20:16.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:16.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:16.254 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:20:16.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:20:16.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:16.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:20:16.411 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:17.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:17.412 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:17.413 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:20:17.413 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:20:17.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:17.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:20:17.572 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2107472045,v1:127.0.0.1:6811/2107472045] [v2:127.0.0.1:6812/2107472045,v1:127.0.0.1:6813/2107472045] exists,up dc2959ac-5628-44c8-bbbc-8ea2a06aa177 2026-02-20T17:20:17.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 2 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:20:17.576 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:20:17.577 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:20:17.578 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:20:17.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:20:17.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16' 2026-02-20T17:20:17.579 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:20:17.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:20:17.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=73f1508a-5ea3-4ca4-99af-dbe14149d16c 2026-02-20T17:20:17.580 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 73f1508a-5ea3-4ca4-99af-dbe14149d16c' 2026-02-20T17:20:17.580 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 73f1508a-5ea3-4ca4-99af-dbe14149d16c 2026-02-20T17:20:17.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:20:17.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRl5hpeI0QIxAA1hT0zDhlSHr4zNPPQM/qzA== 2026-02-20T17:20:17.589 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRl5hpeI0QIxAA1hT0zDhlSHr4zNPPQM/qzA=="}' 2026-02-20T17:20:17.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 73f1508a-5ea3-4ca4-99af-dbe14149d16c -i td/osd-scrub-test/2/new.json 2026-02-20T17:20:17.751 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:20:17.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:20:17.763 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 --mkfs --key AQDRl5hpeI0QIxAA1hT0zDhlSHr4zNPPQM/qzA== --osd-uuid 73f1508a-5ea3-4ca4-99af-dbe14149d16c 2026-02-20T17:20:17.789 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:17.785+0000 7f35b2fc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:17.791 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:17.785+0000 7f35b2fc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:17.792 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:17.789+0000 7f35b2fc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:17.792 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:17.789+0000 7f35b2fc8a40 -1 bdev(0x561eff67ec00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:20:17.793 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:17.789+0000 7f35b2fc8a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:20:18.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:20:18.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:20:18.429 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:20:18.429 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:20:18.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:20:18.634 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:20:18.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:20:18.635 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=15 --osd_scrub_end_hour=16 2026-02-20T17:20:18.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:20:18.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:20:18.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:20:18.658 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:18.653+0000 7f22d343da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:18.660 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:18.657+0000 7f22d343da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:18.661 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:18.657+0000 7f22d343da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:20:18.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:20:18.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:20:18.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:20:18.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:20:18.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:20:18.796 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:18.796 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:20:18.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:20:18.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:18.797 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:20:18.951 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:19.118 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:19.113+0000 7f22d343da40 -1 Falling back to public interface 2026-02-20T17:20:19.757 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:20:19.753+0000 7f22d343da40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:20:19.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:19.952 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:19.953 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:20:19.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:20:19.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:19.953 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:20:20.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:20:21.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:20:21.121 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:20:21.121 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:20:21.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:20:21.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:20:21.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:20:21.281 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3218219389,v1:127.0.0.1:6819/3218219389] [v2:127.0.0.1:6820/3218219389,v1:127.0.0.1:6821/3218219389] exists,up 73f1508a-5ea3-4ca4-99af-dbe14149d16c 2026-02-20T17:20:21.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:20:21.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:20:21.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:20:21.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:435: TEST_scrub_permit_time: create_pool test 1 1 2026-02-20T17:20:21.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:20:21.482 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:20:21.494 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:20:22.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: wait_for_clean 2026-02-20T17:20:22.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:20:22.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:20:22.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:20:22.496 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:20:22.497 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:20:22.497 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:20:22.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:20:22.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:20:22.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:20:22.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:20:22.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:20:22.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:20:22.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:20:22.600 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:20:22.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:20:22.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:20:22.755 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:20:22.755 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:20:22.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:20:22.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:20:22.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:20:22.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-20T17:20:22.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-20T17:20:22.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-20T17:20:22.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:20:22.827 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:20:22.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-20T17:20:22.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-20T17:20:22.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-38654705671' 2026-02-20T17:20:22.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:20:22.900 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:20:22.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T17:20:22.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T17:20:22.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-38654705671 2-60129542148' 2026-02-20T17:20:22.973 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:20:22.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-20T17:20:22.974 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:20:22.975 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:20:22.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-20T17:20:22.976 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:20:22.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-20T17:20:22.978 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-20T17:20:22.978 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836490 2026-02-20T17:20:22.978 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:20:23.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T17:20:23.150 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:20:24.151 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:20:24.151 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:20:24.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-20T17:20:24.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:20:25.312 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:20:25.313 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:20:25.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836490 2026-02-20T17:20:25.473 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:20:25.473 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-20T17:20:25.474 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:20:25.475 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:20:25.475 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-20T17:20:25.476 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:20:25.477 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-20T17:20:25.477 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705671 2026-02-20T17:20:25.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-20T17:20:25.478 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:20:25.638 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705671 2026-02-20T17:20:25.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:20:25.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T17:20:25.639 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:20:25.640 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:20:25.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T17:20:25.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:20:25.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T17:20:25.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T17:20:25.643 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T17:20:25.644 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:20:25.809 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542148 2026-02-20T17:20:25.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:20:25.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:20:25.811 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:20:26.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:20:26.002 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:20:26.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:20:26.002 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:20:26.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:20:26.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:20:26.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:20:26.003 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:20:26.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:20:26.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:20:26.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:20:26.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:20:26.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:20:26.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:20:26.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:20:26.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: get_pg test SOMETHING 2026-02-20T17:20:26.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2026-02-20T17:20:26.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-20T17:20:26.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test SOMETHING 2026-02-20T17:20:26.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-20T17:20:26.511 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: local pgid=1.0 2026-02-20T17:20:26.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: get_primary test SOMETHING 2026-02-20T17:20:26.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:20:26.511 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-20T17:20:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:20:26.512 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test SOMETHING 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: local primary=1 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:26.668 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:26.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: local last_scrub=2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:26.823 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:446: TEST_scrub_permit_time: ceph tell 1.0 schedule-scrub 86400 2026-02-20T17:20:26.888 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:20:26.889 INFO:tasks.workunit.client.0.trial013.stdout: "deep": false, 2026-02-20T17:20:26.889 INFO:tasks.workunit.client.0.trial013.stdout: "must": false, 2026-02-20T17:20:26.889 INFO:tasks.workunit.client.0.trial013.stdout: "stamp": "2026-02-19T17:20:26.888249+0000" 2026-02-20T17:20:26.889 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:20:26.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i=0 )) 2026-02-20T17:20:26.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:26.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:26.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:26.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:26.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:26.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:27.055 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-20T17:20:21.481462+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:27.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:28.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:28.057 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:28.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:28.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:28.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:28.058 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:28.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:28.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-20T17:20:21.481462+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:28.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:29.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:29.213 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:29.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:29.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:29.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:29.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:29.214 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:29.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:29.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:30.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:30.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:30.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:30.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:30.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:30.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:30.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:30.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:30.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:31.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:31.524 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:31.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:31.684 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:32.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:32.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:32.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:32.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:32.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:32.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:32.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:32.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:32.846 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:33.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:33.847 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:33.848 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:33.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:33.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:34.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:34.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:35.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:35.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:35.007 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:35.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:35.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:35.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:35.008 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:35.162 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:35.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:36.163 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:36.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:36.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:36.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:36.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:36.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:36.165 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:36.323 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:37.324 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:37.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:37.325 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:37.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:37.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:37.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:37.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:37.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:37.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:38.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:38.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:38.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:38.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:38.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:38.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:38.488 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:38.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:38.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:39.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:39.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:39.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:39.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:39.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:39.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:39.647 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:39.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:39.806 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:40.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:40.808 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:40.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:40.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:40.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:40.809 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:40.810 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:40.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:40.966 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:41.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:41.968 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:41.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:41.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:41.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:41.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:41.969 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:42.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:42.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:43.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:43.124 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:43.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:43.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:43.125 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:43.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:43.126 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:43.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:43.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:44.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:44.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:44.282 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:44.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:44.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:44.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:44.283 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:44.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:44.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:45.439 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:45.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:45.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:45.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:45.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:45.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:45.441 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:45.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:45.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:46.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:46.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:46.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:46.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:46.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:46.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:46.596 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:46.748 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:46.749 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:47.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:47.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:47.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:47.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:47.750 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:47.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:47.751 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:47.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:47.903 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:48.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:48.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:48.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:48.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:48.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:48.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:48.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:49.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:49.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:50.060 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:50.061 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:50.061 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:50.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:50.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:50.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:50.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:50.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:50.215 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:51.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:51.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:51.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:51.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:51.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:51.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:51.218 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:51.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:52.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:52.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:52.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:52.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:52.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:52.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:52.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:53.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:53.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:53.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:53.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:53.694 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:54.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:54.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:54.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:54.695 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:54.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:54.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:54.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:54.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:55.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:55.851 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:55.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:55.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:55.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:55.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:55.852 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:56.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:56.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:57.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:57.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:57.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:57.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:57.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:58.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:58.175 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:58.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:58.176 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:58.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:58.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:58.177 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:58.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:58.335 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:20:59.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:20:59.336 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:20:59.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:20:59.336 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:20:59.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:20:59.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:20:59.337 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:20:59.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:20:59.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:21:00.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:21:00.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:21:00.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-20T17:21:00.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:00.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:21:00.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:00.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:21:00.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-19T17:20:26.888249+0000 '>' 2026-02-20T17:20:21.481462+0000 2026-02-20T17:21:00.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-20T17:21:01.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-20T17:21:01.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-20T17:21:01.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:455: TEST_scrub_permit_time: perf_counters td/osd-scrub-test 3 2026-02-20T17:21:01.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:21:01.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:21:01.658 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:21:01.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:21:01.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:01.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:21:01.661 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:01.734 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:01.734 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:01.734 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.734 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.735 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.736 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.737 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.738 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:01.739 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:21:01.740 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:01.804 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:01.804 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.805 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.806 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.807 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.808 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:01.809 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:01.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:01.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:21:01.810 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.874 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.875 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_scrub_permit_time ------------------ 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_permit_time ------------------' 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:01.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:01.985 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:01.986 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:21:01.986 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:21:01.987 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:21:01.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:21:01.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:21:01.989 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:21:01.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:01.990 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:21:01.991 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:21:01.991 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:01.992 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:21:01.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:21:01.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:21:02.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:21:02.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.005 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.005 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:21:02.006 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:21:02.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:21:02.007 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_permit_time ------------------' 2026-02-20T17:21:02.007 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_scrub_permit_time ------------------ 2026-02-20T17:21:02.008 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-20T17:21:02.008 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Prepare Test TEST_scrub_test ------------------- 2026-02-20T17:21:02.009 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_test -------------------' 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:02.010 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:02.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:02.011 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:21:02.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:21:02.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:21:02.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:21:02.013 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:21:02.013 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:21:02.014 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:02.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:21:02.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:21:02.015 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:02.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:21:02.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:21:02.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:21:02.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:21:02.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.019 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.019 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:21:02.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:21:02.020 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:21:02.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-20T17:21:02.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:21:02.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.022 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.515468 2026-02-20T17:21:02.024 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:21:02.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-20T17:21:02.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:21:02.025 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-20T17:21:02.025 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Run Test TEST_scrub_test ----------------------- 2026-02-20T17:21:02.026 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_test -----------------------' 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_test td/osd-scrub-test 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:52: TEST_scrub_test: local dir=td/osd-scrub-test 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:53: TEST_scrub_test: local poolname=test 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:54: TEST_scrub_test: local OSDS=3 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:55: TEST_scrub_test: local objects=15 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:57: TEST_scrub_test: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-20T17:21:02.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:21:02.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:21:02.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:21:02.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-20T17:21:02.028 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-20T17:21:02.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:21:02.047 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:02.048 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:02.048 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:02.048 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.048 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.049 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:02.049 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-20T17:21:02.075 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:21:02.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:21:02.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:21:02.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:21:02.076 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:21:02.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:21:02.077 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:21:02.078 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get fsid 2026-02-20T17:21:02.133 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:21:02.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:21:02.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:21:02.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:21:02.134 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:21:02.135 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:21:02.135 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:21:02.135 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:21:02.135 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:21:02.136 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.136 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.136 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.515468/ceph-mon.a.asok 2026-02-20T17:21:02.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:21:02.136 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.515468/ceph-mon.a.asok config get mon_host 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:58: TEST_scrub_test: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-20T17:21:02.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:21:02.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:21:02.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:02.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:02.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:02.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:02.286 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:21:02.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-20T17:21:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-20T17:21:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:60: TEST_scrub_test: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-20T17:21:02.304 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-20T17:21:02.305 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: expr 3 - 1 2026-02-20T17:21:02.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: seq 0 2 2026-02-20T17:21:02.307 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:02.308 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-20T17:21:02.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-20T17:21:02.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:21:02.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3f0b2d72-9365-4deb-a415-548cd26e7d1f 2026-02-20T17:21:02.311 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 3f0b2d72-9365-4deb-a415-548cd26e7d1f 2026-02-20T17:21:02.311 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3f0b2d72-9365-4deb-a415-548cd26e7d1f' 2026-02-20T17:21:02.311 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:21:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+l5hp5sYZExAARPl0QJPXVo54jPaFvDihlg== 2026-02-20T17:21:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+l5hp5sYZExAARPl0QJPXVo54jPaFvDihlg=="}' 2026-02-20T17:21:02.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3f0b2d72-9365-4deb-a415-548cd26e7d1f -i td/osd-scrub-test/0/new.json 2026-02-20T17:21:02.410 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:02.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-20T17:21:02.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQD+l5hp5sYZExAARPl0QJPXVo54jPaFvDihlg== --osd-uuid 3f0b2d72-9365-4deb-a415-548cd26e7d1f 2026-02-20T17:21:02.435 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:02.428+0000 7f803f179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:02.437 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:02.432+0000 7f803f179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:02.437 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:02.432+0000 7f803f179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:02.438 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:02.432+0000 7f803f179a40 -1 bdev(0x556a2604ac00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:21:02.438 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:02.432+0000 7f803f179a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-20T17:21:03.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-20T17:21:03.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:21:03.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:21:03.167 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:21:03.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:21:03.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:21:03.264 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:21:03.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:03.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:21:03.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:21:03.265 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:21:03.278 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:03.272+0000 7f803747aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:03.280 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:03.272+0000 7f803747aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:03.280 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:03.276+0000 7f803747aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:03.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:21:03.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:03.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:21:03.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:03.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:03.357 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:03.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:03.357 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:03.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:03.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:03.445 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:03.690 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:03.684+0000 7f803747aa40 -1 Falling back to public interface 2026-02-20T17:21:04.321 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:04.316+0000 7f803747aa40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:21:04.446 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:04.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:04.447 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:04.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:04.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:04.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:04.561 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:05.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:05.563 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:05.563 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:21:05.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:21:05.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:05.564 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:05.724 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2616512384,v1:127.0.0.1:6801/2616512384] [v2:127.0.0.1:6802/2616512384,v1:127.0.0.1:6803/2616512384] exists,up 3f0b2d72-9365-4deb-a415-548cd26e7d1f 2026-02-20T17:21:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:21:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:21:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:21:05.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:05.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:05.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-20T17:21:05.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-20T17:21:05.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:21:05.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3fdb4e62-fda2-4a8f-8096-2b2d5458de15 2026-02-20T17:21:05.731 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 3fdb4e62-fda2-4a8f-8096-2b2d5458de15 2026-02-20T17:21:05.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3fdb4e62-fda2-4a8f-8096-2b2d5458de15' 2026-02-20T17:21:05.732 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:21:05.741 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQABmJhpG5YgLBAAA/JQoaC6ZfWc27rox+jxQA== 2026-02-20T17:21:05.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQABmJhpG5YgLBAAA/JQoaC6ZfWc27rox+jxQA=="}' 2026-02-20T17:21:05.742 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3fdb4e62-fda2-4a8f-8096-2b2d5458de15 -i td/osd-scrub-test/1/new.json 2026-02-20T17:21:05.904 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:05.914 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-20T17:21:05.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQABmJhpG5YgLBAAA/JQoaC6ZfWc27rox+jxQA== --osd-uuid 3fdb4e62-fda2-4a8f-8096-2b2d5458de15 2026-02-20T17:21:05.931 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:05.924+0000 7f193d87fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:05.933 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:05.928+0000 7f193d87fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:05.934 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:05.928+0000 7f193d87fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:05.934 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:05.928+0000 7f193d87fa40 -1 bdev(0x561095e66c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:21:05.934 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:05.928+0000 7f193d87fa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-20T17:21:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-20T17:21:06.655 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:21:06.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:21:06.656 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:21:06.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:21:06.859 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:21:06.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:21:06.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:06.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:21:06.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:21:06.860 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:21:06.884 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:06.876+0000 7f542d282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:06.886 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:06.880+0000 7f542d282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:06.887 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:06.880+0000 7f542d282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:07.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:21:07.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:07.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:21:07.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:07.015 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:07.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:07.016 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:07.016 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:07.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:07.017 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:21:07.172 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:07.294 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:07.288+0000 7f542d282a40 -1 Falling back to public interface 2026-02-20T17:21:07.914 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:07.908+0000 7f542d282a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:21:08.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:08.173 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:08.174 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:08.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:08.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:08.174 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:21:08.330 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:09.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:09.332 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:09.333 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:21:09.333 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:21:09.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:09.334 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:21:09.491 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/592297261,v1:127.0.0.1:6811/592297261] [v2:127.0.0.1:6812/592297261,v1:127.0.0.1:6813/592297261] exists,up 3fdb4e62-fda2-4a8f-8096-2b2d5458de15 2026-02-20T17:21:09.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:21:09.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:21:09.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:21:09.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:09.495 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:21:09.496 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:09.497 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:21:09.498 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:21:09.499 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 2026-02-20T17:21:09.499 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 2026-02-20T17:21:09.500 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1' 2026-02-20T17:21:09.500 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:21:09.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFmJhpanQhHhAAvBvwlXLezm+h0KGX+ewSxA== 2026-02-20T17:21:09.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFmJhpanQhHhAAvBvwlXLezm+h0KGX+ewSxA=="}' 2026-02-20T17:21:09.507 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 -i td/osd-scrub-test/2/new.json 2026-02-20T17:21:09.666 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:21:09.676 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-20T17:21:09.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAFmJhpanQhHhAAvBvwlXLezm+h0KGX+ewSxA== --osd-uuid 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 2026-02-20T17:21:09.702 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:09.700+0000 7fc8b4479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:09.704 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:09.700+0000 7fc8b4479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:09.705 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:09.704+0000 7fc8b4479a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:09.706 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:09.704+0000 7fc8b4479a40 -1 bdev(0x55b641f72c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:21:09.706 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:09.704+0000 7fc8b4479a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-20T17:21:10.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-20T17:21:10.340 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:21:10.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:21:10.341 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:21:10.341 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:21:10.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:21:10.546 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:21:10.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:10.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:21:10.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:21:10.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:21:10.569 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:10.568+0000 7f1976ef6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:10.571 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:10.568+0000 7f1976ef6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:10.572 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:10.568+0000 7f1976ef6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:10.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:21:10.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:10.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:21:10.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:10.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:10.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:10.705 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:10.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:10.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:10.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:21:10.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:10.966 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:10.964+0000 7f1976ef6a40 -1 Falling back to public interface 2026-02-20T17:21:11.511 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:11.508+0000 7f1976ef6a40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:21:11.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:11.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:11.866 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:11.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:11.866 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:11.867 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:21:12.033 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:13.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:13.034 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:13.035 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:21:13.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:21:13.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:13.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:21:13.193 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/239051219,v1:127.0.0.1:6819/239051219] [v2:127.0.0.1:6820/239051219,v1:127.0.0.1:6821/239051219] exists,up 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 2026-02-20T17:21:13.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:21:13.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:21:13.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:21:13.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:68: TEST_scrub_test: create_pool test 1 1 2026-02-20T17:21:13.194 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-20T17:21:13.438 INFO:tasks.workunit.client.0.trial013.stderr:pool 'test' created 2026-02-20T17:21:13.448 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-20T17:21:14.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:69: TEST_scrub_test: wait_for_clean 2026-02-20T17:21:14.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:14.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:14.450 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:14.451 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:14.451 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:14.451 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:14.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:14.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:14.452 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:14.558 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:14.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:14.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-20T17:21:14.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-20T17:21:14.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-20T17:21:14.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:14.789 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-20T17:21:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-20T17:21:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671' 2026-02-20T17:21:14.862 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:14.862 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T17:21:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T17:21:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671 2-60129542148' 2026-02-20T17:21:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:14.933 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-20T17:21:14.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:14.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:14.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-20T17:21:14.936 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:14.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-20T17:21:14.937 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836491 2026-02-20T17:21:14.937 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-20T17:21:14.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:15.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-20T17:21:15.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:16.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:16.108 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:16.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-20T17:21:16.262 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:17.263 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:21:17.263 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:17.426 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836491 2026-02-20T17:21:17.427 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:17.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-20T17:21:17.427 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:17.428 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:17.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-20T17:21:17.429 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:17.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-20T17:21:17.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-20T17:21:17.431 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705671 2026-02-20T17:21:17.431 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:17.594 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705671 2026-02-20T17:21:17.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:17.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T17:21:17.595 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:17.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:17.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T17:21:17.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:17.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T17:21:17.599 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542148 2026-02-20T17:21:17.599 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T17:21:17.600 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:17.769 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542148 2026-02-20T17:21:17.770 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:17.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:17.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:17.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:17.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:17.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:17.965 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:17.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:17.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:17.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:17.966 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:18.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:18.123 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:18.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:18.124 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:18.320 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:18.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:18.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:18.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: ceph osd dump 2026-02-20T17:21:18.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: grep '^pool.*['\'']test['\'']' 2026-02-20T17:21:18.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: awk '{ print $2 }' 2026-02-20T17:21:18.480 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: poolid=1 2026-02-20T17:21:18.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: file_with_random_data 1032 2026-02-20T17:21:18.481 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=1032 2026-02-20T17:21:18.481 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-20T17:21:18.482 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.483 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.eAchJUMo2a bs=1032 count=1 2026-02-20T17:21:18.484 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:21:18.484 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:21:18.484 INFO:tasks.workunit.client.0.trial013.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 8.9085e-05 s, 11.6 MB/s 2026-02-20T17:21:18.484 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: local testdata_file=/tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: seq 1 15 2026-02-20T17:21:18.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.486 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj1 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.505 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.506 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj2 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.523 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj3 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj4 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.562 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj5 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj6 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.611 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj7 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.633 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj8 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.657 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.658 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj9 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj10 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj11 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj12 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj13 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj14 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-20T17:21:18.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj15 /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.815 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rm -f /tmp/tmp.eAchJUMo2a 2026-02-20T17:21:18.816 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: get_primary test obj1 2026-02-20T17:21:18.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:21:18.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:21:18.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:21:18.817 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:21:18.979 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: local primary=1 2026-02-20T17:21:18.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: get_not_primary test obj1 2026-02-20T17:21:18.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-20T17:21:18.980 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-20T17:21:18.980 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-20T17:21:18.980 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-20T17:21:18.981 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-20T17:21:18.981 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-20T17:21:18.981 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:21:19.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-20T17:21:19.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-20T17:21:19.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-20T17:21:19.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: local otherosd=0 2026-02-20T17:21:19.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:81: TEST_scrub_test: '[' 0 = 2 ']' 2026-02-20T17:21:19.303 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:85: TEST_scrub_test: local anotherosd=2 2026-02-20T17:21:19.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: file_with_random_data 512 2026-02-20T17:21:19.304 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=512 2026-02-20T17:21:19.304 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-20T17:21:19.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.305 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.Qivnd9w6Rn bs=512 count=1 2026-02-20T17:21:19.306 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records in 2026-02-20T17:21:19.306 INFO:tasks.workunit.client.0.trial013.stderr:1+0 records out 2026-02-20T17:21:19.306 INFO:tasks.workunit.client.0.trial013.stderr:512 bytes copied, 8.8394e-05 s, 5.8 MB/s 2026-02-20T17:21:19.306 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: local corrupt_data_file=/tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: objectstore_tool td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.308 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-test 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-test 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-test TERM osd.2 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:19.309 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:19.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:19.310 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-test 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-test/2 2026-02-20T17:21:19.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-test/2 obj1 set-bytes /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:19.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-test 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-test/2 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:19.901 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-20T17:21:19.902 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-test/2 2026-02-20T17:21:19.904 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:21:19.904 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T17:21:19.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=3bd5e6fb-751e-4df3-b39c-e6d3a7c8b84e --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.515468/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-20T17:21:19.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-test/2/whoami 2026-02-20T17:21:19.905 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:21:19.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:21:19.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:21:19.906 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:21:19.924 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:19.919+0000 7fab83ab2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:19.925 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:19.923+0000 7fab83ab2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:19.926 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:19.923+0000 7fab83ab2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:21:20.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:20.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:21:20.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:20.321 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:20.319+0000 7fab83ab2a40 -1 Falling back to public interface 2026-02-20T17:21:20.975 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:20.971+0000 7fab83ab2a40 -1 osd.2 18 log_to_monitors true 2026-02-20T17:21:21.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:21.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-20T17:21:21.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:21.222 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:21.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:21.223 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:21:21.374 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 19 last_clean_interval [14,18) [v2:127.0.0.1:6818/1679792486,v1:127.0.0.1:6819/1679792486] [v2:127.0.0.1:6820/1679792486,v1:127.0.0.1:6821/1679792486] exists,up 447f9120-cf4d-4cbb-9fd0-b8ebc83831a1 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:21.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:21.376 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:21.376 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:21.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:21.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:21.376 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:21.484 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:21.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:21.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:21.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:21.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:21.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:21.645 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:21.646 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:21.646 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:21.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:21.646 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:21.646 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:21.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-20T17:21:21.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-20T17:21:21.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-20T17:21:21.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:21.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:21.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705679 2026-02-20T17:21:21.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705679 2026-02-20T17:21:21.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705679' 2026-02-20T17:21:21.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:21.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:21.863 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-20T17:21:21.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-20T17:21:21.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705679 2-94489280514' 2026-02-20T17:21:21.864 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:21.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-20T17:21:21.864 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:21.865 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:21.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-20T17:21:21.866 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:21.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-20T17:21:21.868 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-20T17:21:21.868 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836499 2026-02-20T17:21:21.868 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:22.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836499 2026-02-20T17:21:22.041 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:23.042 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:23.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:23.201 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-20T17:21:23.202 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:23.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705679 2026-02-20T17:21:23.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:23.203 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:23.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705679 2026-02-20T17:21:23.204 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:23.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705679 2026-02-20T17:21:23.206 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2026-02-20T17:21:23.206 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 38654705679 2026-02-20T17:21:23.206 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:23.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705680 -lt 38654705679 2026-02-20T17:21:23.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:23.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280514 2026-02-20T17:21:23.370 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:23.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:23.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280514 2026-02-20T17:21:23.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:23.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-20T17:21:23.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280514' 2026-02-20T17:21:23.373 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 94489280514 2026-02-20T17:21:23.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:23.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280515 -lt 94489280514 2026-02-20T17:21:23.533 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:23.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:23.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:23.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:23.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:23.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:23.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:23.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:23.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:23.727 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:23.876 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:23.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:23.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:23.877 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:24.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:24.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:24.065 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:24.066 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:90: TEST_scrub_test: rm -f /tmp/tmp.Qivnd9w6Rn 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:92: TEST_scrub_test: local pgid=1.0 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:93: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:24.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:24.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:24.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:24.068 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:24.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:21:24.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:21:24.354 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:21:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:21:24.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:24.355 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:24.508 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:24.508 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:24.508 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:24.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:24.509 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:24.509 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:24.581 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-20T17:21:24.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-20T17:21:24.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-20T17:21:24.582 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:24.582 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:24.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705683 2026-02-20T17:21:24.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705683 2026-02-20T17:21:24.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705683' 2026-02-20T17:21:24.656 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:24.656 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:24.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280518 2026-02-20T17:21:24.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280518 2026-02-20T17:21:24.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705683 2-94489280518' 2026-02-20T17:21:24.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:24.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-20T17:21:24.731 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:24.732 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:24.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-20T17:21:24.733 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:24.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-20T17:21:24.735 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-20T17:21:24.735 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836502 2026-02-20T17:21:24.735 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:24.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-20T17:21:24.896 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:24.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705683 2026-02-20T17:21:24.897 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:24.898 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:24.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705683 2026-02-20T17:21:24.899 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:24.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705683 2026-02-20T17:21:24.901 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705683 2026-02-20T17:21:24.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2026-02-20T17:21:24.901 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:25.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705682 -lt 38654705683 2026-02-20T17:21:25.058 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:26.059 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:26.059 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:26.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705682 -lt 38654705683 2026-02-20T17:21:26.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:27.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:21:27.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:27.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705685 -lt 38654705683 2026-02-20T17:21:27.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:27.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280518 2026-02-20T17:21:27.378 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:27.380 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:27.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280518 2026-02-20T17:21:27.381 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:27.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280518 2026-02-20T17:21:27.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280518' 2026-02-20T17:21:27.383 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280518 2026-02-20T17:21:27.383 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:27.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280518 2026-02-20T17:21:27.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:21:27.555 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:21:27.555 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:21:27.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:21:27.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:21:27.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:21:27.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:21:27.556 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:21:27.630 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:27.631 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:27.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:27.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:27.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:27.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-20T17:21:27.935 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-20T17:21:13.435236+0000 last_deep_scrub_stamp 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:27.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:27.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:27.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:27.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:28.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:13.435236+0000 '>' 2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:28.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:29.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:29.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:29.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:29.109 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:29.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:29.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:29.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:29.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:13.435236+0000 '>' 2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:29.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:30.270 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:30.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:30.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:30.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:30.271 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:30.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:30.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:30.430 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:13.435236+0000 '>' 2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:30.431 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:31.432 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:31.433 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:31.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:31.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:31.433 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:31.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:31.434 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:31.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:28.698089+0000 '>' 2026-02-20T17:21:13.435236+0000 2026-02-20T17:21:31.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:21:31.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: ceph pg dump pgs 2026-02-20T17:21:31.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep '^1.0' 2026-02-20T17:21:31.597 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep -q -- +inconsistent 2026-02-20T17:21:31.744 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:21:31.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:31.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-20T17:21:31.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: test 2 = 2 2026-02-20T17:21:31.828 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: ceph osd out 1 2026-02-20T17:21:32.031 INFO:tasks.workunit.client.0.trial013.stderr:marked out osd.1. 2026-02-20T17:21:32.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: wait_for_clean 2026-02-20T17:21:32.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:32.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:32.044 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:32.044 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:32.045 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:32.045 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:32.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:32.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:32.046 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:32.160 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:32.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:32.321 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:32.321 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:32.321 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:32.322 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:32.322 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:32.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-20T17:21:32.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-20T17:21:32.398 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-20T17:21:32.399 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:32.399 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705692 2026-02-20T17:21:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705692 2026-02-20T17:21:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-38654705692' 2026-02-20T17:21:32.471 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:32.471 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:32.545 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280527 2026-02-20T17:21:32.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280527 2026-02-20T17:21:32.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-38654705692 2-94489280527' 2026-02-20T17:21:32.546 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:32.546 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-20T17:21:32.547 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:32.547 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:32.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-20T17:21:32.548 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:32.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-20T17:21:32.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-20T17:21:32.550 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836511 2026-02-20T17:21:32.550 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:32.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836511 2026-02-20T17:21:32.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:32.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705692 2026-02-20T17:21:32.713 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:32.715 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:32.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705692 2026-02-20T17:21:32.716 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:32.717 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705692 2026-02-20T17:21:32.718 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705692 2026-02-20T17:21:32.718 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2026-02-20T17:21:32.718 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:32.880 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705692 -lt 38654705692 2026-02-20T17:21:32.881 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:32.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280527 2026-02-20T17:21:32.881 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:32.882 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:32.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280527 2026-02-20T17:21:32.883 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:32.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280527 2026-02-20T17:21:32.885 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280527' 2026-02-20T17:21:32.885 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280527 2026-02-20T17:21:32.886 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:33.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280527 -lt 94489280527 2026-02-20T17:21:33.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:33.041 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:33.042 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:33.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:33.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:33.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:33.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:33.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:33.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:33.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:33.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:33.381 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:33.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:33.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:33.382 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:101: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-20T17:21:33.570 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:33.571 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:33.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:33.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:33.572 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:33.849 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:34.011 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:34.011 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:34.012 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:34.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:34.012 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:34.012 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:34.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-20T17:21:34.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-20T17:21:34.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-20T17:21:34.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:34.082 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:34.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705695 2026-02-20T17:21:34.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705695 2026-02-20T17:21:34.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-38654705695' 2026-02-20T17:21:34.155 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:34.155 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:34.226 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280530 2026-02-20T17:21:34.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280530 2026-02-20T17:21:34.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-38654705695 2-94489280530' 2026-02-20T17:21:34.227 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-20T17:21:34.227 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:34.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:34.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-20T17:21:34.229 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-20T17:21:34.231 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-20T17:21:34.231 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836514 2026-02-20T17:21:34.231 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:34.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836514 2026-02-20T17:21:34.393 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:35.394 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:35.394 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:35.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836514 2026-02-20T17:21:35.552 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:35.552 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705695 2026-02-20T17:21:35.553 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:35.553 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:35.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705695 2026-02-20T17:21:35.554 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:35.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705695 2026-02-20T17:21:35.556 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705695 2026-02-20T17:21:35.556 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705695' 2026-02-20T17:21:35.557 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:35.725 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705695 -lt 38654705695 2026-02-20T17:21:35.726 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:35.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280530 2026-02-20T17:21:35.726 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:35.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:35.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280530 2026-02-20T17:21:35.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:35.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280530 2026-02-20T17:21:35.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280530' 2026-02-20T17:21:35.730 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280530 2026-02-20T17:21:35.730 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:35.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280530 -lt 94489280530 2026-02-20T17:21:35.887 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-20T17:21:35.888 INFO:tasks.workunit.client.0.trial013.stdout:#---------- 1.0 loop 0 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-20T17:21:35.889 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+remapped 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+remapped == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:35.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:36.117 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-20T17:21:28.698089+0000 2026-02-20T17:21:36.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-20T17:21:36.263 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.0 to deep-scrub 2026-02-20T17:21:36.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-20T17:21:28.698089+0000 last_deep_scrub_stamp 2026-02-20T17:21:36.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:21:36.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:21:28.698089+0000 2026-02-20T17:21:36.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-20T17:21:36.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:36.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:36.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:28.698089+0000 '>' 2026-02-20T17:21:28.698089+0000 2026-02-20T17:21:36.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:37.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:37.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:37.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:37.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:37.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:37.437 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:37.438 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:37.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:28.698089+0000 '>' 2026-02-20T17:21:28.698089+0000 2026-02-20T17:21:37.595 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:38.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:38.596 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:38.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-20T17:21:38.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:38.597 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-20T17:21:38.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:38.598 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-20T17:21:38.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:36.390954+0000 '>' 2026-02-20T17:21:28.698089+0000 2026-02-20T17:21:38.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:21:38.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:38.756 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-20T17:21:38.827 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: test 2 = 2 2026-02-20T17:21:38.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:38.828 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-20T17:21:38.900 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: test 2 = 2 2026-02-20T17:21:38.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: ceph pg dump pgs 2026-02-20T17:21:38.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep '^1.0' 2026-02-20T17:21:38.901 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep -q -- +inconsistent 2026-02-20T17:21:39.045 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:21:39.056 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: ceph osd in 1 2026-02-20T17:21:39.260 INFO:tasks.workunit.client.0.trial013.stderr:marked in osd.1. 2026-02-20T17:21:39.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: wait_for_clean 2026-02-20T17:21:39.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:39.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:39.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:39.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:39.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:39.276 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:39.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:39.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:39.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:39.385 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:39.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:39.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:39.386 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:39.550 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:39.550 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:39.550 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:39.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:39.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:39.551 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-20T17:21:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-20T17:21:39.625 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520' 2026-02-20T17:21:39.626 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:39.626 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:39.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705701 2026-02-20T17:21:39.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705701 2026-02-20T17:21:39.695 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-38654705701' 2026-02-20T17:21:39.696 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:39.696 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:39.766 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280536 2026-02-20T17:21:39.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280536 2026-02-20T17:21:39.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-38654705701 2-94489280536' 2026-02-20T17:21:39.767 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:39.767 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-20T17:21:39.768 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:39.768 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:39.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836520 2026-02-20T17:21:39.769 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:39.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836520 2026-02-20T17:21:39.771 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-20T17:21:39.771 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836520 2026-02-20T17:21:39.771 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:39.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836520 2026-02-20T17:21:39.941 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:40.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:40.942 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:41.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836521 -lt 21474836520 2026-02-20T17:21:41.102 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:41.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705701 2026-02-20T17:21:41.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:41.104 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:41.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705701 2026-02-20T17:21:41.105 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:41.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705701 2026-02-20T17:21:41.107 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2026-02-20T17:21:41.107 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705701 2026-02-20T17:21:41.107 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:41.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705701 2026-02-20T17:21:41.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:41.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280536 2026-02-20T17:21:41.276 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:41.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:41.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280536 2026-02-20T17:21:41.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:41.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280536 2026-02-20T17:21:41.280 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280536 2026-02-20T17:21:41.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280536' 2026-02-20T17:21:41.280 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:41.447 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280537 -lt 94489280536 2026-02-20T17:21:41.448 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:41.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:41.449 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:41.641 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:41.803 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:41.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:41.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:41.804 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:41.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:110: TEST_scrub_test: repair 1.0 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:41.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:21:42.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:42.159 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-20T17:21:42.306 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-20T17:21:42.317 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:42.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:21:42.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:42.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:21:42.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:42.318 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:21:42.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:36.390954+0000 '>' 2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:42.478 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:43.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:43.479 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:43.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:21:43.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:43.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:21:43.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:43.480 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:21:43.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:36.390954+0000 '>' 2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:43.636 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-20T17:21:44.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-20T17:21:44.637 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-20T17:21:44.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-20T17:21:44.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-20T17:21:44.637 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-20T17:21:44.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-20T17:21:44.638 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-20T17:21:44.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T17:21:43.705972+0000 '>' 2026-02-20T17:21:36.390954+0000 2026-02-20T17:21:44.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-20T17:21:44.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:111: TEST_scrub_test: wait_for_clean 2026-02-20T17:21:44.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:44.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:44.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:44.799 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:44.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:44.799 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:44.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:44.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:44.800 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:44.911 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:44.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:44.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:44.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:44.912 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:44.912 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:45.076 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:45.077 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:45.077 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:45.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:45.077 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:45.077 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:45.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836527 2026-02-20T17:21:45.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836527 2026-02-20T17:21:45.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527' 2026-02-20T17:21:45.147 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:45.147 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:45.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705708 2026-02-20T17:21:45.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705708 2026-02-20T17:21:45.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-38654705708' 2026-02-20T17:21:45.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:45.220 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:45.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280543 2026-02-20T17:21:45.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280543 2026-02-20T17:21:45.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-38654705708 2-94489280543' 2026-02-20T17:21:45.293 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:45.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836527 2026-02-20T17:21:45.294 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:45.295 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:45.295 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836527 2026-02-20T17:21:45.296 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:45.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836527 2026-02-20T17:21:45.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2026-02-20T17:21:45.298 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836527 2026-02-20T17:21:45.298 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:45.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836527 2026-02-20T17:21:45.453 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:46.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:46.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:46.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836527 2026-02-20T17:21:46.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:47.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-20T17:21:47.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:47.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836528 -lt 21474836527 2026-02-20T17:21:47.772 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:47.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705708 2026-02-20T17:21:47.773 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:47.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:47.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705708 2026-02-20T17:21:47.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:47.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705708 2026-02-20T17:21:47.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705708' 2026-02-20T17:21:47.777 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705708 2026-02-20T17:21:47.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:47.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705709 -lt 38654705708 2026-02-20T17:21:47.938 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:47.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280543 2026-02-20T17:21:47.939 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:47.940 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:47.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280543 2026-02-20T17:21:47.941 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:47.942 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280543 2026-02-20T17:21:47.943 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280543 2026-02-20T17:21:47.943 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280543' 2026-02-20T17:21:47.943 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:48.109 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280544 -lt 94489280543 2026-02-20T17:21:48.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:48.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:48.110 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:48.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:48.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:48.299 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:48.454 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:48.454 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:48.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:48.455 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:48.642 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:48.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:48.643 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:48.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:48.643 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-20T17:21:48.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: test 2 = 2 2026-02-20T17:21:48.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: ceph pg dump pgs 2026-02-20T17:21:48.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep '^1.0' 2026-02-20T17:21:48.714 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-20T17:21:48.862 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:21:48.872 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: ceph osd out 1 2026-02-20T17:21:49.076 INFO:tasks.workunit.client.0.trial013.stderr:marked out osd.1. 2026-02-20T17:21:49.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: wait_for_clean 2026-02-20T17:21:49.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:21:49.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:21:49.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:21:49.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:21:49.088 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:21:49.088 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:21:49.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:21:49.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:21:49.089 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:21:49.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:21:49.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:21:49.210 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:21:49.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:21:49.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-20T17:21:49.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:21:49.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:21:49.371 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:21:49.371 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:21:49.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:21:49.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:49.372 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:21:49.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-20T17:21:49.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-20T17:21:49.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-20T17:21:49.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:49.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:21:49.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705713 2026-02-20T17:21:49.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705713 2026-02-20T17:21:49.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713' 2026-02-20T17:21:49.516 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:21:49.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:21:49.585 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280548 2026-02-20T17:21:49.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280548 2026-02-20T17:21:49.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713 2-94489280548' 2026-02-20T17:21:49.586 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:49.586 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-20T17:21:49.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:49.587 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:21:49.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-20T17:21:49.589 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:49.590 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-20T17:21:49.590 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836532 2026-02-20T17:21:49.591 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-20T17:21:49.591 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:49.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836532 2026-02-20T17:21:49.760 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:21:50.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-20T17:21:50.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:21:50.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836532 2026-02-20T17:21:50.916 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:50.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705713 2026-02-20T17:21:50.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:50.917 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:21:50.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705713 2026-02-20T17:21:50.918 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705713 2026-02-20T17:21:50.920 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 38654705713 2026-02-20T17:21:50.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705713' 2026-02-20T17:21:50.921 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:21:51.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705714 -lt 38654705713 2026-02-20T17:21:51.088 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:21:51.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280548 2026-02-20T17:21:51.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:21:51.090 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:21:51.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280548 2026-02-20T17:21:51.091 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:21:51.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280548 2026-02-20T17:21:51.092 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 94489280548 2026-02-20T17:21:51.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280548' 2026-02-20T17:21:51.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:21:51.249 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280549 -lt 94489280548 2026-02-20T17:21:51.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:21:51.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:51.250 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:51.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:21:51.435 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:21:51.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:21:51.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:21:51.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:21:51.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:21:51.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:21:51.436 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:21:51.588 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:21:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:21:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:21:51.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:21:51.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:21:51.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:21:51.774 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:21:51.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:51.775 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-20T17:21:51.844 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: test 0 = 0 2026-02-20T17:21:51.845 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:51.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-20T17:21:51.915 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: test 0 = 0 2026-02-20T17:21:51.916 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: ceph pg 1.0 query 2026-02-20T17:21:51.917 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: jq '.peer_info[1].stats.stat_sum.num_scrub_errors' 2026-02-20T17:21:51.987 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: test 0 = 0 2026-02-20T17:21:51.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: ceph pg dump pgs 2026-02-20T17:21:51.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep '^1.0' 2026-02-20T17:21:51.988 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-20T17:21:52.136 INFO:tasks.workunit.client.0.trial013.stderr:dumped pgs 2026-02-20T17:21:52.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: perf_counters td/osd-scrub-test 3 2026-02-20T17:21:52.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-20T17:21:52.146 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-20T17:21:52.147 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-20T17:21:52.148 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-20T17:21:52.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:52.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-20T17:21:52.149 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:52.219 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.220 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.221 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 2, 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.222 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:52.223 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.224 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:52.225 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:52.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:52.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-20T17:21:52.225 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.290 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.291 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 6, 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.292 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.293 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.294 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:52.295 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:52.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-20T17:21:52.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-20T17:21:52.296 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-20T17:21:52.361 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_ec": [ 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.362 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_dp_repl": [ 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "level": "deep", 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.363 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_ec": [ 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.364 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "ec" 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: ], 2026-02-20T17:21:52.365 INFO:tasks.workunit.client.0.trial013.stdout: "osd_scrub_sh_repl": [ 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: { 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "labels": { 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "level": "shallow", 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "pooltype": "replicated" 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: }, 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "counters": { 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "preemptions": 0, 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_selected": 0, 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "chunk_busy": 0, 2026-02-20T17:21:52.366 INFO:tasks.workunit.client.0.trial013.stdout: "locked_object": 0, 2026-02-20T17:21:52.367 INFO:tasks.workunit.client.0.trial013.stdout: "write_blocked_by_scrub": 0 2026-02-20T17:21:52.367 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.367 INFO:tasks.workunit.client.0.trial013.stdout: } 2026-02-20T17:21:52.367 INFO:tasks.workunit.client.0.trial013.stdout: ] 2026-02-20T17:21:52.367 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:21:52.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_test ------------------' 2026-02-20T17:21:52.368 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Teardown Test TEST_scrub_test ------------------ 2026-02-20T17:21:52.370 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:52.371 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:52.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:52.372 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:52.673 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:52.673 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:21:52.675 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:21:52.675 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:21:52.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:21:52.677 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:21:52.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:21:52.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:21:52.679 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:21:52.680 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.681 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:21:52.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:21:52.682 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:21:52.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:21:52.697 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.698 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:52.698 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:21:52.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:21:52.699 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:21:52.699 INFO:tasks.workunit.client.0.trial013.stdout:-------------- Complete Test TEST_scrub_test ------------------ 2026-02-20T17:21:52.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_test ------------------' 2026-02-20T17:21:52.700 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-test 0 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:52.701 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:52.702 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:52.703 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:21:52.704 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:21:52.704 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:21:52.705 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:21:52.706 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:21:52.706 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:21:52.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.707 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:21:52.708 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:21:52.708 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.709 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:21:52.710 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T17:21:52.711 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-20T17:21:52.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:21:52.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.712 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.515468 2026-02-20T17:21:52.712 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.515468 2026-02-20T17:21:52.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:21:52.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:21:52.713 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T17:21:52.714 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T17:21:52.715 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T17:21:52.723 INFO:tasks.workunit:Running workunit scrub/osd-unexpected-clone.sh... 2026-02-20T17:21:52.724 DEBUG:teuthology.orchestra.run.trial013:workunit test scrub/osd-unexpected-clone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2026-02-20T17:21:52.773 INFO:tasks.workunit.client.0.trial013.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-20T17:21:52.777 INFO:tasks.workunit.client.0.trial013.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-20T17:21:52.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-unexpected-clone 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:22: run: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:23: run: shift 2026-02-20T17:21:52.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: export CEPH_MON=127.0.0.1:7144 2026-02-20T17:21:52.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: CEPH_MON=127.0.0.1:7144 2026-02-20T17:21:52.779 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:26: run: export CEPH_ARGS 2026-02-20T17:21:52.779 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: uuidgen 2026-02-20T17:21:52.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: CEPH_ARGS+='--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none ' 2026-02-20T17:21:52.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-20T17:21:52.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:30: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-20T17:21:52.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: set 2026-02-20T17:21:52.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-20T17:21:52.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: local funcs=TEST_recover_unexpected 2026-02-20T17:21:52.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:32: run: for func in $funcs 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:33: run: setup td/osd-unexpected-clone 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-unexpected-clone 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-20T17:21:52.782 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:21:52.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:21:52.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:21:52.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:21:52.783 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:21:52.784 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:21:52.784 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:21:52.785 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:21:52.785 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:21:52.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:21:52.786 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:21:52.787 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:21:52.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:21:52.788 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:21:52.788 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:21:52.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:21:52.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:21:52.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-20T17:21:52.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:21:52.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:52.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.663321 2026-02-20T17:21:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:21:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:21:52.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-unexpected-clone 2026-02-20T17:21:52.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-20T17:21:52.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.795 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:52.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.663321 2026-02-20T17:21:52.797 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-unexpected-clone 1' TERM HUP INT 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:34: run: TEST_recover_unexpected td/osd-unexpected-clone 2026-02-20T17:21:52.798 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:40: TEST_recover_unexpected: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:42: TEST_recover_unexpected: run_mon td/osd-unexpected-clone a 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-unexpected-clone/a 2026-02-20T17:21:52.799 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-unexpected-clone/a --run-dir=td/osd-unexpected-clone 2026-02-20T17:21:52.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-20T17:21:52.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:52.821 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:52.822 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:52.822 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.822 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:52.822 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:52.822 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-unexpected-clone/a '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --mon-cluster-log-file=td/osd-unexpected-clone/log --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-20T17:21:52.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-20T17:21:52.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-20T17:21:52.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:21:52.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:21:52.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-20T17:21:52.851 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.853 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:52.854 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.663321/ceph-mon.a.asok 2026-02-20T17:21:52.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:21:52.854 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.663321/ceph-mon.a.asok config get fsid 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-20T17:21:52.909 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-20T17:21:52.910 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:52.910 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:52.910 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.663321/ceph-mon.a.asok 2026-02-20T17:21:52.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-20T17:21:52.910 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.663321/ceph-mon.a.asok config get mon_host 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:43: TEST_recover_unexpected: run_mgr td/osd-unexpected-clone x 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-unexpected-clone 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-unexpected-clone/x 2026-02-20T17:21:52.965 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:53.062 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:53.063 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:21:53.063 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-unexpected-clone/x '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-20T17:21:53.082 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:44: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 0 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/0 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:53.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/0' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/0/journal' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:53.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:53.085 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:21:53.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:21:53.086 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/0 2026-02-20T17:21:53.086 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:21:53.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e05104f5-aa38-447f-a41e-1e2007164672 2026-02-20T17:21:53.087 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e05104f5-aa38-447f-a41e-1e2007164672' 2026-02-20T17:21:53.087 INFO:tasks.workunit.client.0.trial013.stdout:add osd0 e05104f5-aa38-447f-a41e-1e2007164672 2026-02-20T17:21:53.088 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:21:53.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxmJhpliODBRAAdCV8A6SJHLyYddg6Fe+B2w== 2026-02-20T17:21:53.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxmJhpliODBRAAdCV8A6SJHLyYddg6Fe+B2w=="}' 2026-02-20T17:21:53.094 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e05104f5-aa38-447f-a41e-1e2007164672 -i td/osd-unexpected-clone/0/new.json 2026-02-20T17:21:53.182 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:53.190 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/0/new.json 2026-02-20T17:21:53.191 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxmJhpliODBRAAdCV8A6SJHLyYddg6Fe+B2w== --osd-uuid e05104f5-aa38-447f-a41e-1e2007164672 2026-02-20T17:21:53.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:53.203+0000 7f87c5006a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:53.207 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:53.203+0000 7f87c5006a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:53.208 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:53.207+0000 7f87c5006a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:53.209 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:53.207+0000 7f87c5006a40 -1 bdev(0x563775376c00 td/osd-unexpected-clone/0/block) open stat got: (1) Operation not permitted 2026-02-20T17:21:53.209 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:53.207+0000 7f87c5006a40 -1 bluestore(td/osd-unexpected-clone/0) _read_fsid unparsable uuid 2026-02-20T17:21:53.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/0/keyring 2026-02-20T17:21:53.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:21:53.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-20T17:21:53.994 INFO:tasks.workunit.client.0.trial013.stdout:adding osd0 key to auth repository 2026-02-20T17:21:53.994 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:21:54.099 INFO:tasks.workunit.client.0.trial013.stdout:start osd.0 2026-02-20T17:21:54.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-20T17:21:54.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:21:54.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:21:54.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:21:54.100 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:21:54.124 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:54.119+0000 7f3d6489fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:54.126 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:54.123+0000 7f3d6489fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:54.127 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:54.123+0000 7f3d6489fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:54.192 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:54.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:54.193 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:54.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:54.542 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:54.539+0000 7f3d6489fa40 -1 Falling back to public interface 2026-02-20T17:21:55.243 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:55.239+0000 7f3d6489fa40 -1 osd.0 0 log_to_monitors true 2026-02-20T17:21:55.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:55.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:21:55.287 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:55.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:55.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:55.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:55.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:56.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:56.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:21:56.378 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:21:56.378 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:21:56.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:56.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-20T17:21:56.535 INFO:tasks.workunit.client.0.trial013.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3556343854,v1:127.0.0.1:6801/3556343854] [v2:127.0.0.1:6802/3556343854,v1:127.0.0.1:6803/3556343854] exists,up e05104f5-aa38-447f-a41e-1e2007164672 2026-02-20T17:21:56.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:21:56.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:21:56.538 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:45: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 1 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-20T17:21:56.539 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:21:56.540 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:21:56.541 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:21:56.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-20T17:21:56.542 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:21:56.542 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:21:56.543 INFO:tasks.workunit.client.0.trial013.stdout:add osd1 fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:21:56.543 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fd28e390-5e7b-47e1-b567-d120a051faa7' 2026-02-20T17:21:56.543 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:21:56.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0mJhp4j/EIBAAXTAQOxWs7KIwgfLNlmaU2w== 2026-02-20T17:21:56.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0mJhp4j/EIBAAXTAQOxWs7KIwgfLNlmaU2w=="}' 2026-02-20T17:21:56.551 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd28e390-5e7b-47e1-b567-d120a051faa7 -i td/osd-unexpected-clone/1/new.json 2026-02-20T17:21:56.711 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:56.722 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/1/new.json 2026-02-20T17:21:56.723 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0mJhp4j/EIBAAXTAQOxWs7KIwgfLNlmaU2w== --osd-uuid fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:21:56.746 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:56.743+0000 7f3f08bb6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:56.747 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:56.743+0000 7f3f08bb6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:56.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:56.747+0000 7f3f08bb6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:56.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:56.747+0000 7f3f08bb6a40 -1 bdev(0x5614ef87ac00 td/osd-unexpected-clone/1/block) open stat got: (1) Operation not permitted 2026-02-20T17:21:56.749 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:56.747+0000 7f3f08bb6a40 -1 bluestore(td/osd-unexpected-clone/1) _read_fsid unparsable uuid 2026-02-20T17:21:57.436 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/1/keyring 2026-02-20T17:21:57.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:21:57.437 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-20T17:21:57.438 INFO:tasks.workunit.client.0.trial013.stdout:adding osd1 key to auth repository 2026-02-20T17:21:57.438 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:21:57.638 INFO:tasks.workunit.client.0.trial013.stdout:start osd.1 2026-02-20T17:21:57.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-20T17:21:57.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:21:57.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:21:57.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:21:57.639 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:21:57.653 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:57.651+0000 7f0526476a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:57.655 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:57.651+0000 7f0526476a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:57.656 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:57.651+0000 7f0526476a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:57.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:21:57.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:21:58.086 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:58.082+0000 7f0526476a40 -1 Falling back to public interface 2026-02-20T17:21:58.738 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:21:58.734+0000 7f0526476a40 -1 osd.1 0 log_to_monitors true 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:21:58.957 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:21:59.120 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:00.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:00.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:00.122 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:22:00.122 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:22:00.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:00.123 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:22:00.280 INFO:tasks.workunit.client.0.trial013.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1835183380,v1:127.0.0.1:6811/1835183380] [v2:127.0.0.1:6812/1835183380,v1:127.0.0.1:6813/1835183380] exists,up fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:22:00.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:00.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:00.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:46: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 2 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-20T17:22:00.286 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-20T17:22:00.287 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-20T17:22:00.287 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:00.287 INFO:tasks.workunit.client.0.trial013.stdout:add osd2 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:00.288 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 84502525-3e0a-4de0-be25-132910a3ec81' 2026-02-20T17:22:00.288 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-20T17:22:00.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4mJhpxhypERAAz90rG/N9xV/mCra2R5cWbA== 2026-02-20T17:22:00.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4mJhpxhypERAAz90rG/N9xV/mCra2R5cWbA=="}' 2026-02-20T17:22:00.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 84502525-3e0a-4de0-be25-132910a3ec81 -i td/osd-unexpected-clone/2/new.json 2026-02-20T17:22:00.459 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:22:00.469 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/2/new.json 2026-02-20T17:22:00.470 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4mJhpxhypERAAz90rG/N9xV/mCra2R5cWbA== --osd-uuid 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:00.486 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:00.482+0000 7f7f719b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:00.488 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:00.482+0000 7f7f719b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:00.488 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:00.486+0000 7f7f719b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:00.489 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:00.486+0000 7f7f719b9a40 -1 bdev(0x55d500610c00 td/osd-unexpected-clone/2/block) open stat got: (1) Operation not permitted 2026-02-20T17:22:00.489 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:00.486+0000 7f7f719b9a40 -1 bluestore(td/osd-unexpected-clone/2) _read_fsid unparsable uuid 2026-02-20T17:22:01.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/2/keyring 2026-02-20T17:22:01.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-20T17:22:01.093 INFO:tasks.workunit.client.0.trial013.stdout:adding osd2 key to auth repository 2026-02-20T17:22:01.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-20T17:22:01.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-20T17:22:01.296 INFO:tasks.workunit.client.0.trial013.stdout:start osd.2 2026-02-20T17:22:01.297 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-20T17:22:01.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:01.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-20T17:22:01.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-20T17:22:01.298 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-20T17:22:01.312 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:01.306+0000 7f89a5d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:01.313 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:01.310+0000 7f89a5d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:01.315 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:01.310+0000 7f89a5d3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:01.451 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:01.452 INFO:tasks.workunit.client.0.trial013.stdout:0 2026-02-20T17:22:01.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:01.452 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:01.613 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:01.710 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:01.706+0000 7f89a5d3da40 -1 Falling back to public interface 2026-02-20T17:22:02.358 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:02.354+0000 7f89a5d3da40 -1 osd.2 0 log_to_monitors true 2026-02-20T17:22:02.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:02.614 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:02.615 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:02.615 INFO:tasks.workunit.client.0.trial013.stdout:1 2026-02-20T17:22:02.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:02.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:02.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:03.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:03.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:03.777 INFO:tasks.workunit.client.0.trial013.stdout:2 2026-02-20T17:22:03.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-20T17:22:03.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:03.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:03.938 INFO:tasks.workunit.client.0.trial013.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2684499534,v1:127.0.0.1:6819/2684499534] [v2:127.0.0.1:6820/2684499534,v1:127.0.0.1:6821/2684499534] exists,up 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:03.939 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:48: TEST_recover_unexpected: ceph osd pool create foo 1 2026-02-20T17:22:04.141 INFO:tasks.workunit.client.0.trial013.stderr:pool 'foo' created 2026-02-20T17:22:04.153 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:49: TEST_recover_unexpected: rados -p foo put foo /etc/passwd 2026-02-20T17:22:05.046 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:50: TEST_recover_unexpected: rados -p foo mksnap snap 2026-02-20T17:22:05.094 INFO:tasks.workunit.client.0.trial013.stdout:created pool foo snap snap 2026-02-20T17:22:05.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:51: TEST_recover_unexpected: rados -p foo put foo /etc/group 2026-02-20T17:22:05.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:53: TEST_recover_unexpected: wait_for_clean 2026-02-20T17:22:05.118 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:05.119 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:05.120 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:05.228 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:05.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:05.386 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:05.386 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:05.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:05.386 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:05.387 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:05.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-20T17:22:05.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-20T17:22:05.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-20T17:22:05.458 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:05.459 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:05.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-20T17:22:05.530 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-20T17:22:05.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-20T17:22:05.531 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:05.531 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:05.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-20T17:22:05.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-20T17:22:05.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-20T17:22:05.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:05.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-20T17:22:05.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:05.604 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:05.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-20T17:22:05.605 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:05.606 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-20T17:22:05.607 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-20T17:22:05.607 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.0 seq 21474836483 2026-02-20T17:22:05.607 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:05.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T17:22:05.775 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:06.776 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:06.777 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:06.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-20T17:22:06.935 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:07.936 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:22:07.937 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-20T17:22:08.095 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:08.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-20T17:22:08.096 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:08.097 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:08.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-20T17:22:08.098 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:08.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-20T17:22:08.099 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-20T17:22:08.099 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.1 seq 42949672963 2026-02-20T17:22:08.100 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:08.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-20T17:22:08.269 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:08.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-20T17:22:08.270 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:08.271 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:08.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-20T17:22:08.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:08.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-20T17:22:08.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-20T17:22:08.274 INFO:tasks.workunit.client.0.trial013.stdout:waiting osd.2 seq 60129542146 2026-02-20T17:22:08.274 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:08.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-20T17:22:08.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:08.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:08.444 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:08.631 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:08.632 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:08.782 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:08.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:08.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:08.783 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:08.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:08.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:08.972 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:08.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: get_primary foo foo 2026-02-20T17:22:08.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=foo 2026-02-20T17:22:08.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=foo 2026-02-20T17:22:08.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map foo foo 2026-02-20T17:22:08.973 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-20T17:22:09.136 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: local osd=1 2026-02-20T17:22:09.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 --op list foo 2026-02-20T17:22:09.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-20T17:22:09.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: grep snapid.:1 2026-02-20T17:22:09.138 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 --op list foo 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:22:09.139 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:22:09.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:22:09.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:22:09.140 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 --op list foo 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-20T17:22:09.445 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 --op list foo 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-20T17:22:09.619 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:09.620 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:22:09.621 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:22:09.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-20T17:22:09.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T17:22:09.622 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T17:22:09.622 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:09.623 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-20T17:22:09.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T17:22:09.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:22:09.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:22:09.623 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:22:09.636 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:09.630+0000 7f90b66b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:09.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:09.634+0000 7f90b66b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:09.638 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:09.634+0000 7f90b66b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:09.780 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:09.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:09.781 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:22:09.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:09.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:22:09.944 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:09.986 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:09.982+0000 7f90b66b7a40 -1 Falling back to public interface 2026-02-20T17:22:10.637 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:10.634+0000 7f90b66b7a40 -1 osd.1 17 log_to_monitors true 2026-02-20T17:22:10.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:10.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:10.946 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:10.946 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:10.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:10.947 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:22:11.102 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6810/2186619143,v1:127.0.0.1:6811/2186619143] [v2:127.0.0.1:6812/2186619143,v1:127.0.0.1:6813/2186619143] exists,up fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:11.103 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:11.104 INFO:tasks.workunit.client.0.trial013.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:11.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:11.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:11.104 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:11.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:11.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:11.201 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:11.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:11.202 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:11.202 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:11.364 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:11.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-20T17:22:11.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-20T17:22:11.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-20T17:22:11.435 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:11.436 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:11.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313218 2026-02-20T17:22:11.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313218 2026-02-20T17:22:11.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-90194313218' 2026-02-20T17:22:11.503 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:11.503 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:11.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-20T17:22:11.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-20T17:22:11.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-90194313218 2-60129542148' 2026-02-20T17:22:11.576 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:11.577 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-20T17:22:11.577 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:11.577 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:11.578 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-20T17:22:11.578 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:11.579 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-20T17:22:11.580 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-20T17:22:11.580 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836486 2026-02-20T17:22:11.580 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:11.746 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-20T17:22:11.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:12.747 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:12.748 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:12.905 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-20T17:22:12.906 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:13.907 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:22:13.907 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:14.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-20T17:22:14.065 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:14.065 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-90194313218 2026-02-20T17:22:14.065 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:14.066 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:14.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-90194313218 2026-02-20T17:22:14.067 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:14.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313218 2026-02-20T17:22:14.069 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 90194313218' 2026-02-20T17:22:14.069 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 90194313218 2026-02-20T17:22:14.069 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:14.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313218 -lt 90194313218 2026-02-20T17:22:14.230 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:14.231 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-20T17:22:14.231 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:14.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:14.232 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-20T17:22:14.233 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:14.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-20T17:22:14.234 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-20T17:22:14.234 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542148 2026-02-20T17:22:14.234 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:14.398 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-20T17:22:14.399 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:14.399 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:14.399 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:14.587 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:14.588 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:14.737 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:14.737 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:14.737 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:14.737 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:14.929 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:14.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:14.930 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:14.930 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: JSON='["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:22:14.931 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:58: TEST_recover_unexpected: echo 'JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-20T17:22:14.931 INFO:tasks.workunit.client.0.trial013.stdout:JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:59: TEST_recover_unexpected: rm -f td/osd-unexpected-clone/_ td/osd-unexpected-clone/data 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:60: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:22:14.933 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:22:14.934 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:22:15.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:22:15.241 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-20T17:22:15.242 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-20T17:22:15.414 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-20T17:22:15.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:15.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:22:15.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-20T17:22:15.415 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:15.416 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:15.417 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:15.418 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:15.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:15.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:15.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:15.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:15.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:start osd.1 2026-02-20T17:22:15.420 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:15.421 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-20T17:22:15.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-20T17:22:15.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:22:15.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:22:15.421 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:22:15.425 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:15.422+0000 7feb3aa98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:15.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:15.422+0000 7feb3aa98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:15.427 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:15.422+0000 7feb3aa98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:15.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-20T17:22:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-20T17:22:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:15.574 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:15.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:15.575 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:22:15.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:15.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:22:15.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:15.878 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:15.874+0000 7feb3aa98a40 -1 Falling back to public interface 2026-02-20T17:22:16.470 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:16.466+0000 7feb3aa98a40 -1 osd.1 23 log_to_monitors true 2026-02-20T17:22:16.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:16.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:16.730 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:16.730 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:16.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:16.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-20T17:22:16.891 INFO:tasks.workunit.client.0.trial013.stderr:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [21,23) [v2:127.0.0.1:6810/2532063749,v1:127.0.0.1:6811/2532063749] [v2:127.0.0.1:6812/2532063749,v1:127.0.0.1:6813/2532063749] exists,up fd28e390-5e7b-47e1-b567-d120a051faa7 2026-02-20T17:22:16.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:16.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:16.893 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:16.893 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:16.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:16.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:16.893 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:16.981 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:16.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:16.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:16.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:16.982 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:16.982 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:17.140 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:17.141 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:17.141 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:17.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:17.141 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:17.141 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:17.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-20T17:22:17.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-20T17:22:17.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-20T17:22:17.211 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:17.211 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:17.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116994 2026-02-20T17:22:17.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116994 2026-02-20T17:22:17.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-115964116994' 2026-02-20T17:22:17.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:17.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:17.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-20T17:22:17.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-20T17:22:17.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-115964116994 2-60129542150' 2026-02-20T17:22:17.352 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:17.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-20T17:22:17.353 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:17.353 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:17.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-20T17:22:17.354 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:17.355 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-20T17:22:17.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-20T17:22:17.356 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836488 2026-02-20T17:22:17.356 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:17.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-20T17:22:17.522 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:17.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116994 2026-02-20T17:22:17.523 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:17.524 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:17.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116994 2026-02-20T17:22:17.525 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116994 2026-02-20T17:22:17.526 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116994' 2026-02-20T17:22:17.526 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 115964116994 2026-02-20T17:22:17.527 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:17.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116994 -lt 115964116994 2026-02-20T17:22:17.683 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:17.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-20T17:22:17.684 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:17.685 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:17.685 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-20T17:22:17.686 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:17.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-20T17:22:17.687 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-20T17:22:17.687 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 60129542150 2026-02-20T17:22:17.687 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:17.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542150 2026-02-20T17:22:17.857 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:18.858 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:18.859 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:19.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542150 2026-02-20T17:22:19.021 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:20.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:22:20.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:20.180 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-20T17:22:20.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:20.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:20.181 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:20.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:20.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:20.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:20.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:20.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:20.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:20.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:20.375 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:20.533 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:20.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:20.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:20.534 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:20.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:20.727 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:61: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-20T17:22:20.728 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:22:20.729 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:22:20.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:22:20.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:22:20.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:22:21.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:21.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:21.216 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-20T17:22:21.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:21.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:22:21.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:22:21.217 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:21.218 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:21.219 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:22:21.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:22:21.221 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:22:21.222 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:22:21.232 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:21.226+0000 7f857a04ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:21.233 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:21.230+0000 7f857a04ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:21.234 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:21.230+0000 7f857a04ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:21.374 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:22:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:22:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:21.375 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:21.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:21.376 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:22:21.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:21.376 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:21.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:21.586 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:21.582+0000 7f857a04ca40 -1 Falling back to public interface 2026-02-20T17:22:22.206 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:22.202+0000 7f857a04ca40 -1 osd.2 29 log_to_monitors true 2026-02-20T17:22:22.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:22.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:22.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:22.534 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:22.534 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:22.535 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:22.689 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [14,29) [v2:127.0.0.1:6818/2676131162,v1:127.0.0.1:6819/2676131162] [v2:127.0.0.1:6820/2676131162,v1:127.0.0.1:6821/2676131162] exists,up 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:22.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:22.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:22.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:22.689 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:22:22.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:22.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:22.690 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:22.690 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:22.691 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:22.691 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:22.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:22.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:22.691 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:22.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:22.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:22.780 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:22.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:22.781 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:22.781 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:22.948 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:22.948 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:22.949 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:22.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:22.949 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:22.949 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-20T17:22:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-20T17:22:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-20T17:22:23.022 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:23.023 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:23.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-20T17:22:23.092 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-20T17:22:23.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-115964116996' 2026-02-20T17:22:23.093 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:23.093 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953474 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953474 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-115964116996 2-137438953474' 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-20T17:22:23.164 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:23.166 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:23.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-20T17:22:23.166 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:23.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-20T17:22:23.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-20T17:22:23.168 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836490 2026-02-20T17:22:23.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:23.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-20T17:22:23.326 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:24.328 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:24.328 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:24.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836490 2026-02-20T17:22:24.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:24.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116996 2026-02-20T17:22:24.490 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:24.491 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:24.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116996 2026-02-20T17:22:24.492 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:24.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116996 2026-02-20T17:22:24.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116996' 2026-02-20T17:22:24.493 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 115964116996 2026-02-20T17:22:24.493 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:24.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-20T17:22:24.660 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:24.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-137438953474 2026-02-20T17:22:24.660 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:24.662 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:24.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-137438953474 2026-02-20T17:22:24.662 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:24.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953474 2026-02-20T17:22:24.663 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 137438953474' 2026-02-20T17:22:24.664 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 137438953474 2026-02-20T17:22:24.664 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:24.830 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953474 -lt 137438953474 2026-02-20T17:22:24.830 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:24.831 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:25.027 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:25.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:25.028 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:25.187 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:25.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:25.187 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:25.188 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:25.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:25.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:25.382 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:25.383 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:63: TEST_recover_unexpected: rados -p foo rmsnap snap 2026-02-20T17:22:25.438 INFO:tasks.workunit.client.0.trial013.stdout:removed pool foo snap snap 2026-02-20T17:22:25.440 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:65: TEST_recover_unexpected: sleep 5 2026-02-20T17:22:30.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:67: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:30.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-20T17:22:30.442 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:22:30.443 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:22:30.444 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:30.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-20T17:22:31.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-20T17:22:31.035 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-20T17:22:31.036 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:31.037 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:31.039 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:22:31.040 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:22:31.055 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:31.050+0000 7f4d49fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:31.056 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:31.054+0000 7f4d49fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:31.057 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:31.054+0000 7f4d49fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:22:31.197 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:31.198 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:31.356 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:31.426 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:31.422+0000 7f4d49fd4a40 -1 Falling back to public interface 2026-02-20T17:22:32.052 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:32.046+0000 7f4d49fd4a40 -1 osd.2 35 log_to_monitors true 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:32.358 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:32.514 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 36 last_clean_interval [32,35) [v2:127.0.0.1:6818/2944594692,v1:127.0.0.1:6819/2944594692] [v2:127.0.0.1:6820/2944594692,v1:127.0.0.1:6821/2944594692] exists,up 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:32.514 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:32.515 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:32.516 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:32.516 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:32.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:32.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:32.516 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:32.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:32.616 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:32.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:32.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:32.617 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:32.617 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:32.777 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:32.778 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:32.778 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:32.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:32.778 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:32.778 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:32.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-20T17:22:32.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-20T17:22:32.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-20T17:22:32.850 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:32.850 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:32.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-20T17:22:32.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-20T17:22:32.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-115964116999' 2026-02-20T17:22:32.923 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:32.923 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:32.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-20T17:22:32.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-20T17:22:32.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-115964116999 2-167503724546' 2026-02-20T17:22:32.995 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:32.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-20T17:22:32.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:32.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:32.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-20T17:22:32.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:32.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-20T17:22:32.998 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-20T17:22:32.998 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836493 2026-02-20T17:22:32.999 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:33.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-20T17:22:33.167 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:34.168 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:34.168 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:34.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-20T17:22:34.325 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:34.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116999 2026-02-20T17:22:34.326 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:34.327 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:34.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116999 2026-02-20T17:22:34.327 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:34.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-20T17:22:34.329 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116999' 2026-02-20T17:22:34.329 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 115964116999 2026-02-20T17:22:34.330 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:34.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-20T17:22:34.485 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:34.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-167503724546 2026-02-20T17:22:34.485 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:34.487 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:34.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-167503724546 2026-02-20T17:22:34.487 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:34.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-20T17:22:34.489 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 167503724546' 2026-02-20T17:22:34.489 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 167503724546 2026-02-20T17:22:34.489 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:34.653 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-20T17:22:34.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:34.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:34.653 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:34.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:34.845 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:34.846 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:34.996 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:34.996 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:34.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:34.997 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:35.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:35.184 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:68: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-20T17:22:35.185 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:22:35.186 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:22:35.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:22:35.492 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:35.493 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-20T17:22:35.789 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-20T17:22:35.790 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-20T17:22:35.791 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-20T17:22:35.792 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-20T17:22:35.793 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:start osd.2 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=7cf2b7f9-70f4-4528-9f6a-1818c5abe72c --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.663321/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-20T17:22:35.794 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-20T17:22:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-20T17:22:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-20T17:22:35.795 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-20T17:22:35.807 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:35.802+0000 7f0d66172a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:35.808 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:35.806+0000 7f0d66172a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:35.809 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:35.806+0000 7f0d66172a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-20T17:22:35.946 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:0 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:35.947 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:36.106 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-20T17:22:36.186 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:36.182+0000 7f0d66172a40 -1 Falling back to public interface 2026-02-20T17:22:36.846 INFO:tasks.workunit.client.0.trial013.stderr:2026-02-20T17:22:36.842+0000 7f0d66172a40 -1 osd.2 40 log_to_monitors true 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-20T17:22:37.108 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-20T17:22:37.267 INFO:tasks.workunit.client.0.trial013.stderr:osd.2 up in weight 1 up_from 44 up_thru 0 down_at 41 last_clean_interval [39,40) [v2:127.0.0.1:6818/2160869521,v1:127.0.0.1:6819/2160869521] [v2:127.0.0.1:6820/2160869521,v1:127.0.0.1:6821/2160869521] exists,up 84502525-3e0a-4de0-be25-132910a3ec81 2026-02-20T17:22:37.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-20T17:22:37.267 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-20T17:22:37.268 INFO:tasks.workunit.client.0.trial013.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-20T17:22:37.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-20T17:22:37.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-20T17:22:37.269 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-20T17:22:37.368 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-20T17:22:37.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-20T17:22:37.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-20T17:22:37.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-20T17:22:37.369 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-20T17:22:37.369 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr:1 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr:2' 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:37.532 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-20T17:22:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836495 2026-02-20T17:22:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836495 2026-02-20T17:22:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495' 2026-02-20T17:22:37.603 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:37.604 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-20T17:22:37.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117001 2026-02-20T17:22:37.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117001 2026-02-20T17:22:37.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495 1-115964117001' 2026-02-20T17:22:37.678 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-20T17:22:37.678 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-20T17:22:37.750 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561026 2026-02-20T17:22:37.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561026 2026-02-20T17:22:37.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495 1-115964117001 2-188978561026' 2026-02-20T17:22:37.751 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:37.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836495 2026-02-20T17:22:37.752 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:37.752 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-20T17:22:37.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836495 2026-02-20T17:22:37.753 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:37.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836495 2026-02-20T17:22:37.754 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2026-02-20T17:22:37.754 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.0 seq 21474836495 2026-02-20T17:22:37.755 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:37.920 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836495 2026-02-20T17:22:37.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:38.921 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-20T17:22:38.922 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:39.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836495 2026-02-20T17:22:39.083 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-20T17:22:40.084 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-20T17:22:40.084 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-20T17:22:40.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836495 2026-02-20T17:22:40.244 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:40.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964117001 2026-02-20T17:22:40.245 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:40.246 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-20T17:22:40.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964117001 2026-02-20T17:22:40.247 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:40.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117001 2026-02-20T17:22:40.248 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964117001' 2026-02-20T17:22:40.248 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.1 seq 115964117001 2026-02-20T17:22:40.248 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-20T17:22:40.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117001 -lt 115964117001 2026-02-20T17:22:40.406 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-20T17:22:40.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-188978561026 2026-02-20T17:22:40.407 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-20T17:22:40.408 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-20T17:22:40.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-188978561026 2026-02-20T17:22:40.409 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-20T17:22:40.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=188978561026 2026-02-20T17:22:40.410 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 188978561026' 2026-02-20T17:22:40.410 INFO:tasks.workunit.client.0.trial013.stderr:waiting osd.2 seq 188978561026 2026-02-20T17:22:40.410 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-20T17:22:40.570 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561026 -lt 188978561026 2026-02-20T17:22:40.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-20T17:22:40.570 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:40.571 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:40.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-20T17:22:40.761 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:40.762 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:40.919 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:22:40.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:40.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:40.920 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:22:41.111 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-20T17:22:41.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:22:41.212 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:41.213 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:41.373 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:22:41.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:41.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:41.374 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:41.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:22:41.567 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T17:22:41.568 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T17:22:41.755 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T17:22:41.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T17:22:41.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-20T17:22:41.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:22:41.756 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-20T17:22:41.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:22:41.956 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:41.957 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:42.113 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:22:42.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:42.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:42.113 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T17:22:42.302 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T17:22:42.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T17:22:42.303 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T17:22:42.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T17:22:42.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T17:22:42.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-20T17:22:42.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:22:42.490 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-20T17:22:42.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:22:42.891 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:42.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:42.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:42.891 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:42.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:42.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:42.892 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:43.045 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-20T17:22:43.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:43.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:43.045 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-20T17:22:43.232 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-20T17:22:43.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-20T17:22:43.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-20T17:22:43.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-20T17:22:43.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-20T17:22:43.419 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-20T17:22:44.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-20T17:22:44.220 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-20T17:22:44.221 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-20T17:22:44.379 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-20T17:22:44.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-20T17:22:44.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-20T17:22:44.380 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-20T17:22:44.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-20T17:22:44.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-20T17:22:44.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-20T17:22:44.573 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:70: TEST_recover_unexpected: sleep 5 2026-02-20T17:22:49.575 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:72: TEST_recover_unexpected: ceph pg repair 1.0 2026-02-20T17:22:49.720 INFO:tasks.workunit.client.0.trial013.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-20T17:22:49.729 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:74: TEST_recover_unexpected: sleep 10 2026-02-20T17:22:59.731 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:76: TEST_recover_unexpected: ceph log last 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426008+0000 osd.1 (osd.1) 2 : cluster [ERR] 1.0 shard 0 1:602f83fe:::foo:1 : missing 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426012+0000 osd.1 (osd.1) 3 : cluster [ERR] 1.0 shard 1 1:602f83fe:::foo:1 : missing 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426098+0000 osd.1 (osd.1) 4 : cluster [ERR] repair 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426226+0000 osd.1 (osd.1) 5 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426242+0000 osd.1 (osd.1) 6 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:50.426264+0000 osd.1 (osd.1) 7 : cluster [ERR] 1.0 repair 3 errors, 2 fixed 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377063+0000 osd.1 (osd.1) 9 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : candidate size 0 info size 15900 mismatch 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377119+0000 osd.1 (osd.1) 10 : cluster [ERR] 1.0 shard 0 soid 1:602f83fe:::foo:1 : data_digest 0x9f81759d != data_digest 0x5af226ef from auth oi 1:602f83fe:::foo:1(17'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]) 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377120+0000 osd.1 (osd.1) 11 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : data_digest 0xffffffff != data_digest 0x9f81759d from shard 0, data_digest 0xffffffff != data_digest 0x5af226ef from auth oi 1:602f83fe:::foo:1(17'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]), size 0 != size 15900 from auth oi 1:602f83fe:::foo:1(17'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]), size 0 != size 15900 from shard 0 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377135+0000 osd.1 (osd.1) 12 : cluster [ERR] 1.0 soid 1:602f83fe:::foo:1 : data_digest 0x5af226ef != data_digest 0x9f81759d from shard 0 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377226+0000 osd.1 (osd.1) 13 : cluster [ERR] deep-scrub 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377431+0000 osd.1 (osd.1) 14 : cluster [ERR] 1.0 deep-scrub 0 missing, 1 inconsistent objects 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:51.377433+0000 osd.1 (osd.1) 15 : cluster [ERR] 1.0 deep-scrub 4 errors 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:53.337522+0000 mon.a (mon.0) 251 : cluster [ERR] Health check failed: 4 scrub errors (OSD_SCRUB_ERRORS) 2026-02-20T17:22:59.910 INFO:tasks.workunit.client.0.trial013.stdout:2026-02-20T17:22:53.337538+0000 mon.a (mon.0) 252 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-02-20T17:22:59.918 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:79: TEST_recover_unexpected: timeout 60 ceph tell osd.0 version 2026-02-20T17:22:59.985 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:22:59.985 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T17:22:59.985 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T17:22:59.985 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T17:22:59.985 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:22:59.993 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:80: TEST_recover_unexpected: timeout 60 ceph tell osd.1 version 2026-02-20T17:23:00.058 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:23:00.058 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T17:23:00.058 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T17:23:00.058 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T17:23:00.059 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:23:00.068 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:81: TEST_recover_unexpected: timeout 60 ceph tell osd.2 version 2026-02-20T17:23:00.133 INFO:tasks.workunit.client.0.trial013.stdout:{ 2026-02-20T17:23:00.134 INFO:tasks.workunit.client.0.trial013.stdout: "version": "20.2.0-677-gf7870454", 2026-02-20T17:23:00.134 INFO:tasks.workunit.client.0.trial013.stdout: "release": "tentacle", 2026-02-20T17:23:00.134 INFO:tasks.workunit.client.0.trial013.stdout: "release_type": "stable" 2026-02-20T17:23:00.134 INFO:tasks.workunit.client.0.trial013.stdout:} 2026-02-20T17:23:00.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:35: run: teardown td/osd-unexpected-clone 2026-02-20T17:23:00.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-20T17:23:00.144 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:23:00.145 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:23:00.254 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:23:00.254 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:23:00.255 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:23:00.255 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:23:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:23:00.257 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:23:00.258 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:23:00.258 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:23:00.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:23:00.259 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:23:00.259 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:23:00.260 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:23:00.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-20T17:23:00.261 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-20T17:23:00.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:23:00.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:23:00.272 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:23:00.272 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.663321 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-unexpected-clone 0 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-20T17:23:00.274 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-20T17:23:00.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-20T17:23:00.275 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-20T17:23:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-20T17:23:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-20T17:23:00.275 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-20T17:23:00.276 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-20T17:23:00.277 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-20T17:23:00.277 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-20T17:23:00.278 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-20T17:23:00.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-20T17:23:00.279 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-20T17:23:00.279 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-20T17:23:00.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:23:00.280 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-20T17:23:00.281 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-20T17:23:00.281 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T17:23:00.282 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:23:00.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-20T17:23:00.283 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-20T17:23:00.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-20T17:23:00.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-20T17:23:00.284 INFO:tasks.workunit.client.0.trial013.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.663321 2026-02-20T17:23:00.284 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.663321 2026-02-20T17:23:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-20T17:23:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-20T17:23:00.285 INFO:tasks.workunit.client.0.trial013.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-20T17:23:00.286 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T17:23:00.287 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-20T17:23:00.341 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2026-02-20T17:23:00.341 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-20T17:23:00.694 DEBUG:teuthology.parallel:result is None 2026-02-20T17:23:00.694 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-20T17:23:00.703 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-20T17:23:00.703 DEBUG:teuthology.orchestra.run.trial013:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-20T17:23:00.748 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-20T17:23:00.748 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T17:23:00.762 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-20T17:23:00.763 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T17:23:00.896 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-20T17:23:00.908 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-20T17:23:00.908 DEBUG:teuthology.orchestra.run.trial013:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout: remote refid st t when poll reach delay offset jitter 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout:============================================================================== 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout:+s2-b.time.mci1. 42.20.202.230 2 u 60 64 377 29.794 -1.113 0.732 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout:-t2.time.bf1.yah 98.139.133.62 2 u 36 64 377 82.599 +34.577 2.193 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout:+chi3.us.ntp.li 204.9.54.119 2 u 63 64 377 19.017 +0.018 0.437 2026-02-20T17:23:00.978 INFO:teuthology.orchestra.run.trial013.stdout:*pool-96-231-54- .E810. 1 u 54 64 377 10.865 -1.546 1.424 2026-02-20T17:23:00.979 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-20T17:23:00.991 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-20T17:23:00.991 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-20T17:23:01.004 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-20T17:23:01.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-20T17:23:01.033 INFO:teuthology.task.internal:Duration was 3500.478822 seconds 2026-02-20T17:23:01.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-20T17:23:01.053 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-20T17:23:01.053 DEBUG:teuthology.orchestra.run.trial013:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T17:23:01.080 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-20T17:23:01.080 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial013.front.sepia.ceph.com 2026-02-20T17:23:01.080 DEBUG:teuthology.orchestra.run.trial013:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T17:23:01.126 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-20T17:23:01.126 DEBUG:teuthology.orchestra.run.trial013:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T17:23:01.392 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-20T17:23:01.392 DEBUG:teuthology.orchestra.run.trial013:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T17:23:01.401 INFO:teuthology.orchestra.run.trial013.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T17:23:01.402 INFO:teuthology.orchestra.run.trial013.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T17:23:01.403 INFO:teuthology.orchestra.run.trial013.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T17:23:01.403 INFO:teuthology.orchestra.run.trial013.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T17:23:01.403 INFO:teuthology.orchestra.run.trial013.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T17:23:01.469 INFO:teuthology.orchestra.run.trial013.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T17:23:01.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-20T17:23:01.488 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-20T17:23:01.489 DEBUG:teuthology.orchestra.run.trial013:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T17:23:01.521 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-20T17:23:01.541 DEBUG:teuthology.orchestra.run.trial013:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:23:01.572 INFO:teuthology.orchestra.run.trial013.stdout:kernel.core_pattern = core 2026-02-20T17:23:01.580 DEBUG:teuthology.orchestra.run.trial013:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T17:23:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T17:23:01.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-20T17:23:01.645 INFO:teuthology.task.internal:Transferring archived files... 2026-02-20T17:23:01.648 DEBUG:teuthology.misc:Transferring archived files from trial013:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:06:38-rados-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60841/remote/trial013 2026-02-20T17:23:01.650 DEBUG:teuthology.orchestra.run.trial013:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T17:23:01.708 INFO:teuthology.task.internal:Removing archive directory... 2026-02-20T17:23:01.709 DEBUG:teuthology.orchestra.run.trial013:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T17:23:01.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-20T17:23:01.755 INFO:teuthology.task.internal:Not uploading archives. 2026-02-20T17:23:01.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-20T17:23:01.772 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-20T17:23:01.772 DEBUG:teuthology.orchestra.run.trial013:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T17:23:01.780 INFO:teuthology.orchestra.run.trial013.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 17:23 /home/ubuntu/cephtest 2026-02-20T17:23:01.781 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-20T17:23:01.792 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-20T17:23:01.827 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 3500.478821516037 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-20T17:23:01.827 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T17:23:01.934 INFO:teuthology.run:pass